Low-complexity sign prediction for video coding

ABSTRACT

A device includes processing circuitry configured to identify a set of pre-stored residual samples in video data that is stored to a memory, to combine the set of stored residual samples to form a combination, to apply a first sign combination to transform coefficients of the combination to obtain a first hypothesis reconstruction for a current block of the video data, and to apply a second sign combination to the transform coefficients of the combination to obtain a second hypothesis reconstruction for the current block. The processing circuitry is further configured to derive respective cost functions with respect to the first and second hypothesis reconstructions, to compare the respective derived cost functions, to select, based on the comparison, either the first or second hypothesis reconstruction, and to code the current block using a sign prediction associated with the selected one of the first or second hypothesis reconstruction.

This application claims the benefit of:

U.S. Provisional Application No. 62/437,642, filed on 21 Dec. 2016;

U.S. Provisional Application No. 62/476,375, filed on 24 Mar. 2017; and

U.S. Provisional Application No. 62/549,336, filed on 23 Aug. 2017, theentire contents of each of which are incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates to video encoding and video decoding.

BACKGROUND

Digital video capabilities can be incorporated into a wide range ofdevices, including digital televisions, digital direct broadcastsystems, wireless broadcast systems, personal digital assistants (PDAs),laptop or desktop computers, tablet computers, e-book readers, digitalcameras, digital recording devices, digital media players, video gamingdevices, video game consoles, cellular or satellite radio telephones,so-called “smart phones,” video teleconferencing devices, videostreaming devices, and the like. Digital video devices implement videocoding techniques, such as those described in the standards defined byMPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Advanced VideoCoding (AVC), the High Efficiency Video Coding (HEVC) standard, andextensions of such standards. The video devices may transmit, receive,encode, decode, and/or store digital video information more efficientlyby implementing such video coding techniques. Video coding standardsinclude ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IECMPEG-2 Visual ITU-T H.263, ISO/IEC MPEG-4 Visual and ITU-T H.264 (alsoknown as ISO/IEC MPEG-4 AVC), including its Scalable Video Coding (SVC)and Multi-view Video Coding (MVC) extensions. In addition, a new videocoding standard, namely High Efficiency Video Coding (HEVC), hasrecently been developed by the Joint Collaboration Team on Video Coding(JCT-VC) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC MotionPicture Experts Group (MPEG). The latest HEVC draft specification, andreferred to as HEVC WD hereinafter, is available fromhttp://phenix.int-evry.fr/jct/doc_end_user/documents/14_Vienna/wg11/JCTVC-N1003-v1.zip.

Video coding techniques include spatial (intra-picture) predictionand/or temporal (inter-picture) prediction to reduce or removeredundancy inherent in video sequences. For block-based video coding, avideo slice (e.g., a video frame or a portion of a video frame) may bepartitioned into video blocks, which may also be referred to astreeblocks, coding units (CUs) and/or coding nodes. Pictures may bereferred to as frames, and reference pictures may be referred to asreference frames.

Spatial or temporal prediction results in a predictive block for a blockto be coded. Residual data represents pixel differences between theoriginal block to be coded and the predictive block. For furthercompression, the residual data may be transformed from the pixel domainto a transform domain, resulting in residual transform coefficients,which then may be quantized. Entropy coding may be applied to achieveeven more compression.

SUMMARY

In an example, this disclosure is directed to a method of coding videodata. The method includes identifying a set of pre-stored residualsamples in a pre-stored portion of the video data, combining the set ofpre-stored residual samples to form a combination of pre-stored residualsamples, applying a first sign combination to transform coefficients ofthe combination of pre-stored residual samples to obtain a firsthypothesis reconstruction with respect to a current block of the videodata, and applying a second sign combination to the transformcoefficients of the combination of pre-stored residual samples to obtaina second hypothesis reconstruction with respect to the current block ofthe video data. The method further includes deriving respective costfunctions with respect to the first hypothesis reconstruction and thesecond hypothesis reconstruction, comparing the respective derived costfunctions, selecting, based on the comparison, either the firsthypothesis reconstruction or the second hypothesis reconstruction, andcoding the current block using a sign prediction associated with theselected one of the first hypothesis reconstruction or the secondhypothesis reconstruction.

In another example, this disclosure is directed to a device for codingdata. The device includes a video data memory, and processing circuitrycoupled to the video data memory. The processing circuitry is configuredto identify a set of pre-stored residual samples in a portion of thevideo data that is pre-stored to the video data memory, to combine theset of pre-stored residual samples to form a combination of pre-storedresidual samples, to apply a first sign combination to transformcoefficients of the combination of pre-stored residual samples to obtaina first hypothesis reconstruction with respect to a current block of thevideo data, and to apply a second sign combination to the transformcoefficients of the combination of pre-stored residual samples to obtaina second hypothesis reconstruction with respect to the current block ofthe video data. The processing circuitry is further configured to deriverespective cost functions with respect to the first hypothesisreconstruction and the second hypothesis reconstruction, to compare therespective derived cost functions, to select, based on the comparison,either the first hypothesis reconstruction or the second hypothesisreconstruction, and to code the current block using a sign predictionassociated with the selected one of the first hypothesis reconstructionor the second hypothesis reconstruction.

In another example, this disclosure is directed to an apparatus forcoding video data. The apparatus includes means for identifying a set ofpre-stored residual samples in a pre-stored portion of the video data,means for combining the set of pre-stored residual samples to form acombination of pre-stored residual samples, means for applying a firstsign combination to transform coefficients of the combination ofpre-stored residual samples to obtain a first hypothesis reconstructionwith respect to a current block of the video data, and means forapplying a second sign combination to the transform coefficients of thecombination of pre-stored residual samples to obtain a second hypothesisreconstruction with respect to the current block of the video data. Theapparatus further includes means for deriving respective cost functionswith respect to the first hypothesis reconstruction and the secondhypothesis reconstruction, means for comparing the respective derivedcost functions, means for selecting, based on the comparison, either thefirst hypothesis reconstruction or the second hypothesis reconstruction,and means for coding the current block using a sign predictionassociated with the selected one of the first hypothesis reconstructionor the second hypothesis reconstruction.

In another example, this disclosure is directed to a non-transitorycomputer-readable medium encoded with instructions that, when executed,cause one or more processors of a device to identify a set of pre-storedresidual samples in a portion of the video data that is pre-stored tothe non-transitory computer-readable storage medium, to combine the setof pre-stored residual samples to form a combination of pre-storedresidual samples, to apply a first sign combination to transformcoefficients of the combination of pre-stored residual samples to obtaina first hypothesis reconstruction with respect to a current block of thevideo data, and to apply a second sign combination to the transformcoefficients of the combination of pre-stored residual samples to obtaina second hypothesis reconstruction with respect to the current block ofthe video data. The instructions, when executed, further cause the oneor more processors to derive respective cost functions with respect tothe first hypothesis reconstruction and the second hypothesisreconstruction, to compare the respective derived cost functions, toselect, based on the comparison, either the first hypothesisreconstruction or the second hypothesis reconstruction, and to code thecurrent block using a sign prediction associated with the selected oneof the first hypothesis reconstruction or the second hypothesisreconstruction.

In another example, this disclosure is directed to a method of codingvideo data. The method includes determining that a block of the videodata is eligible to be coded using sign prediction, performing the signprediction with respect to one or more transform coefficients of theblock only if the block meets a predetermined criterion, where thepredetermined criterion is based on at least one of: a width of theblock, a height of the block, a coding mode used to code the block, or aposition of one or more sign-predicted transform coefficients within theblock, and coding the block based on the sign prediction performed withrespect to the block.

In another example, this disclosure is directed to a device for codingdata. The device includes a memory configured to store video data, andprocessing circuitry coupled to the memory. The processing circuitry isconfigured to determine that a block of the video data stored to thememory is eligible to be coded using sign prediction, to perform thesign prediction with respect to one or more transform coefficients ofthe block only if the block meets a predetermined criterion, where thepredetermined criterion is based on at least one of: a width of theblock, a height of the block, a coding mode used to code the block, or aposition of one or more sign-predicted transform coefficients within theblock, and to code the block based on the sign prediction performed withrespect to the block.

In another example, this disclosure is directed to an apparatus forcoding video data. The apparatus includes means for determining that ablock of video data is eligible to be coded using sign prediction, meansfor performing the sign prediction with respect to one or more transformcoefficients of the block only if the block meets a predeterminedcriterion, where the predetermined criterion is based on at least oneof: a width of the block, a height of the block, a coding mode used tocode the block, or a position of one or more sign-predicted transformcoefficients within the block, and means for coding the block based onthe sign prediction performed with respect to the block.

In another example, this disclosure is directed to a non-transitorycomputer-readable medium that is encoded with instructions. Theinstructions, when executed, cause one or more processors of a device todetermine that a block of video data stored to the non-transitorycomputer-readable storage medium is eligible to be coded using signprediction, to perform the sign prediction with respect to one or moretransform coefficients of the block only if the block meets apredetermined criterion, wherein the predetermined criterion is based onat least one of: a width of the block, a height of the block, a codingmode used to code the block, or a position of one or more sign-predictedtransform coefficients within the block, and to code the block based onthe sign prediction performed with respect to the block.

In an example, this disclosure is directed to a method of coding videodata. The method includes applying a first sign combination to transformcoefficients of a block of the video data to obtain a first hypothesisreconstruction with respect to the block of the video data, applying asecond sign combination to the transform coefficients of the block ofthe video data to obtain a second hypothesis reconstruction with respectto the block of the video data, measuring a first cost function withrespect to the first hypothesis reconstruction by determining a firstset of pixel value gradients that represent respective differencesbetween samples of the first hypothesis reconstruction of the block andsamples of a neighboring block that is positioned adjacent to the block,and measuring a second cost function with respect to the secondhypothesis reconstruction by determining a second set of pixel valuegradients that represent respective differences between samples of thesecond hypothesis reconstruction of the block and samples of theneighboring block that is positioned adjacent to the block. The methodfurther includes comparing the first cost function to the second costfunction, selecting, based on the comparison, either the firsthypothesis reconstruction or the second hypothesis reconstruction, andcoding the block using the respective sign combination associated withthe selected one of the first hypothesis reconstruction or the secondhypothesis reconstruction.

In another example, this disclosure is directed to a device for codingdata. The device includes a memory configured to store video data, andprocessing circuitry in communication with the memory. The processingcircuitry is configured to apply a first sign combination to transformcoefficients of a block of the video data stored to the memory to obtaina first hypothesis reconstruction with respect to the block of the videodata, to apply a second sign combination to the transform coefficientsof the block of the video data stored to the memory to obtain a secondhypothesis reconstruction with respect to the block of the video data,to measure a first cost function with respect to the first hypothesisreconstruction by determining a first set of pixel value gradients thatrepresent respective differences between samples of the first hypothesisreconstruction of the block and samples of a neighboring block that ispositioned adjacent to the block, and to measure a second cost functionwith respect to the second hypothesis reconstruction by determining asecond set of pixel value gradients that represent respectivedifferences between samples of the second hypothesis reconstruction ofthe block and samples of the neighboring block that is positionedadjacent to the block. The processing circuitry is further configured tocompare the first cost function to the second cost function, to select,based on the comparison, either the first hypothesis reconstruction orthe second hypothesis reconstruction, and to code the block using therespective sign combination associated with the selected one of thefirst hypothesis reconstruction or the second hypothesis reconstruction.

In another example, this disclosure is directed to an apparatus forcoding video data. The apparatus includes means for applying a firstsign combination to transform coefficients of a block of video data toobtain a first hypothesis reconstruction with respect to the block ofthe video data, means for applying a second sign combination to thetransform coefficients of the block of the video data to obtain a secondhypothesis reconstruction with respect to the block of the video data,means for measuring a first cost function with respect to the firsthypothesis reconstruction by determining a first set of pixel valuegradients that represent respective differences between samples of thefirst hypothesis reconstruction of the block and samples of aneighboring block that is positioned adjacent to the block, and meansfor measuring a second cost function with respect to the secondhypothesis reconstruction by determining a second set of pixel valuegradients that represent respective differences between samples of thesecond hypothesis reconstruction of the block and samples of theneighboring block that is positioned adjacent to the block. Theapparatus further includes means for comparing the first cost functionto the second cost function, means for selecting, based on thecomparison, either the first hypothesis reconstruction or the secondhypothesis reconstruction; and means for coding the block using therespective sign combination associated with the selected one of thefirst hypothesis reconstruction or the second hypothesis reconstruction.

In another example, this disclosure is directed to a non-transitorycomputer-readable medium encoded with instructions. The instructions,when executed, cause one or more processors of a video coding device toapply a first sign combination to transform coefficients of a block of evideo data stored to the non-transitory computer-readable storage mediumto obtain a first hypothesis reconstruction with respect to the block ofthe video data, to apply a second sign combination to the transformcoefficients of the block of the video data stored to the non-transitorycomputer-readable storage medium to obtain a second hypothesisreconstruction with respect to the block of the video data, to measure afirst cost function with respect to the first hypothesis reconstructionby determining a first set of pixel value gradients that representrespective differences between samples of the first hypothesisreconstruction of the block and samples of a neighboring block that ispositioned adjacent to the block, and to measure a second cost functionwith respect to the second hypothesis reconstruction by determining asecond set of pixel value gradients that represent respectivedifferences between samples of the second hypothesis reconstruction ofthe block and samples of the neighboring block that is positionedadjacent to the block. The instructions, when executed, further causethe one or more processors of the video coding device to compare thefirst cost function to the second cost function, to select, based on thecomparison, either the first hypothesis reconstruction or the secondhypothesis reconstruction, and to code the block using the respectivesign combination associated with the selected one of the firsthypothesis reconstruction or the second hypothesis reconstruction.

The details of one or more examples of the disclosure are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages will be apparent from the description, drawings,and the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example video encoding anddecoding system configured to implement techniques of the disclosure.

FIG. 2 is a conceptual diagram illustrating an example coefficient scanbased on coefficient groups.

FIGS. 3A and 3B are diagrams illustrating hypothesis reconstruction andprediction techniques.

FIG. 4 is a block diagram illustrating an example of a video encoderconfigured to implement techniques of the disclosure.

FIG. 5 is a block diagram illustrating an example of a video decoderconfigured to implement techniques of the disclosure.

FIG. 6 is a flowchart illustrating an example process by which a videocoding device may perform template-based reconstruction techniques ofthis disclosure as part of a hypothesis reconstruction for a currentblock.

FIG. 7 is a flowchart illustrating an example process by which a videocoding device may perform selective sign prediction techniques of thisdisclosure as part of coding a current block of video data.

FIG. 8 is a flowchart illustrating an example process by which a videocoding device may perform cost function measurement techniques of thisdisclosure as part of a hypothesis reconstruction for a current block.

FIG. 9 shows an example of a system for predicting signs at a videodecoder.

FIG. 10 shows an example of a system for predicting signs at a videodecoder using a maximum-likelihood according to the techniques of thisdisclosure.

FIG. 11 shows examples of variable weights in a block.

FIG. 12 shows an example of techniques for determining a set of weightsfor likelihood computation.

DETAILED DESCRIPTION

This disclosure is related to techniques for transform coefficient signbit prediction for video encoding and decoding, as applied for Intra orInter coded blocks, as well as blocks coded in other modes. Aspects ofthis disclosure may be used in the context of advanced video codecs,such as extensions of HEVC or the next generation of video codingstandards.

ITU-T VCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC 29/WG 11) are studying thepotential for standardization of future video coding technology with acompression capability that significantly exceeds that of the currentHEVC standard (including its current extensions and near-term extensionsfor screen content coding and high-dynamic-range coding). The groups areworking together on this exploration activity in a joint collaborationeffort known as the Joint Video Exploration Team (JVET) to evaluatecompression technology designs proposed by the groups in this area. TheJVET first met during 19-21 Oct. 2015. To alleviate various codeccomplexity issues, several optimization methods are designed foralgorithm acceleration in the joint exploration model (JEM). The JEM isa video codec platform for the exploration of next-generation videocoding technologies. Further aspects of the JEM are described in“Algorithm Description of Joint Exploration Test Model 2” Joint VideoExploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG11, 2nd Meeting: San Diego, USA, 20-26 Feb. 2016, Document No.JVET-B1001_v1. A version of the reference software, i.e., JointExploration Model 3 (JEM 3) can be downloaded from:https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/tags/HM-16.6-JEM-3.0/.An algorithm description of Joint Exploration Test Model 3 (JEM3) may bereferred to JVET-C1001.

FIG. 1 is a block diagram illustrating an example video encoding anddecoding system 10 that may be configured to perform techniques of thisdisclosure. As shown in FIG. 1, system 10 includes a source device 12that provides encoded video data to be decoded at a later time by adestination device 14. In particular, source device 12 provides thevideo data to destination device 14 via a computer-readable medium 16.Source device 12 and destination device 14 may comprise any of a widerange of devices, including desktop computers, notebook (i.e., laptop)computers, tablet computers, set-top boxes, telephone handsets such asso-called “smart” phones, tablet computers, televisions, cameras,display devices, digital media players, video gaming consoles, videostreaming device, or the like. In some cases, source device 12 anddestination device 14 may be equipped for wireless communication. Thus,source device 12 and destination device 14 may be wireless communicationdevices. Source device 12 is an example video encoding device (i.e., adevice for encoding video data). Destination device 14 is an examplevideo decoding device (i.e., a device for decoding video data).

In the example of FIG. 1, source device 12 includes a video source 18, astorage medium 20 configured to store video data, a video encoder 22,and an output interface 24. Destination device 14 includes an inputinterface 26, a storage medium 28 configured to store encoded videodata, a video decoder 30, and display device 32. In other examples,source device 12 and destination device 14 include other components orarrangements. For example, source device 12 may receive video data froman external video source, such as an external camera. Likewise,destination device 14 may interface with an external display device,rather than including an integrated display device 32.

The illustrated system 10 of FIG. 1 is merely one example. Techniquesfor processing and/or coding video data may be performed by any digitalvideo encoding and/or decoding device. Although generally the techniquesof this disclosure are performed by a video encoding device or videodecoding device, the techniques may also be performed by a videoencoder/decoder, typically referred to as a “CODEC.” Source device 12and destination device 14 are merely examples of such coding devices inwhich source device 12 generates coded video data for transmission todestination device 14. In some examples, source device 12 anddestination device 14 may operate in a substantially symmetrical mannersuch that each of source device 12 and destination device 14 includevideo encoding and decoding components. Hence, system 10 may supportone-way or two-way video transmission between source device 12 anddestination device 14, e.g., for video streaming, video playback, videobroadcasting, or video telephony.

Video source 18 of source device 12 may include a video capture device,such as a video camera, a video archive containing previously capturedvideo, and/or a video feed interface to receive video data from a videocontent provider. As a further alternative, video source 18 may generatecomputer graphics-based data as the source video, or a combination oflive video, archived video, and computer-generated video. Source device12 may comprise one or more data storage media (e.g., storage media 20)configured to store the video data. The techniques described in thisdisclosure may be applicable to video coding in general, and may beapplied to wireless and/or wired applications. In each case, thecaptured, pre-captured, or computer-generated video may be encoded byvideo encoder 22. Output interface 24 may output the encoded videoinformation to a computer-readable medium 16.

Destination device 14 may receive the encoded video data to be decodedvia computer-readable medium 16. Computer-readable medium 16 maycomprise any type of medium or device capable of moving the encodedvideo data from source device 12 to destination device 14. In someexamples, computer-readable medium 16 comprises a communication mediumto enable source device 12 to transmit encoded video data directly todestination device 14 in real-time. The encoded video data may bemodulated according to a communication standard, such as a wirelesscommunication protocol, and transmitted to destination device 14. Thecommunication medium may comprise any wireless or wired communicationmedium, such as a radio frequency (RF) spectrum or one or more physicaltransmission lines. The communication medium may form part of apacket-based network, such as a local area network, a wide-area network,or a global network such as the Internet. The communication medium mayinclude routers, switches, base stations, or any other equipment thatmay be useful to facilitate communication from source device 12 todestination device 14. Destination device 14 may comprise one or moredata storage media configured to store encoded video data and decodedvideo data.

In some examples, encoded data may be output from output interface 24 toa storage device. Similarly, encoded data may be accessed from thestorage device by input interface. The storage device may include any ofa variety of distributed or locally accessed data storage media such asa hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile ornon-volatile memory, or any other suitable digital storage media forstoring encoded video data. In a further example, the storage device maycorrespond to a file server or another intermediate storage device thatmay store the encoded video generated by source device 12. Destinationdevice 14 may access stored video data from the storage device viastreaming or download. The file server may be any type of server capableof storing encoded video data and transmitting that encoded video datato the destination device 14. Example file servers include a web server(e.g., for a website), an FTP server, network attached storage (NAS)devices, or a local disk drive. Destination device 14 may access theencoded video data through any standard data connection, including anInternet connection. This may include a wireless channel (e.g., a Wi-Ficonnection), a wired connection (e.g., DSL, cable modem, etc.), or acombination of both that is suitable for accessing encoded video datastored on a file server. The transmission of encoded video data from thestorage device may be a streaming transmission, a download transmission,or a combination thereof.

The techniques described in this disclosure may be applied to videocoding in support of any of a variety of multimedia applications, suchas over-the-air television broadcasts, cable television transmissions,satellite television transmissions, Internet streaming videotransmissions, such as dynamic adaptive streaming over HTTP (DASH),digital video that is encoded onto a data storage medium, decoding ofdigital video stored on a data storage medium, or other applications. Insome examples, system 10 may be configured to support one-way or two-wayvideo transmission to support applications such as video streaming,video playback, video broadcasting, and/or video telephony.

Computer-readable medium 16 may include transient media, such as awireless broadcast or wired network transmission, or storage media (thatis, non-transitory storage media), such as a hard disk, flash drive,compact disc, digital video disc, Blu-ray disc, or othercomputer-readable media. In some examples, a network server (not shown)may receive encoded video data from source device 12 and provide theencoded video data to destination device 14, e.g., via networktransmission. Similarly, a computing device of a medium productionfacility, such as a disc stamping facility, may receive encoded videodata from source device 12 and produce a disc containing the encodedvideo data. Therefore, computer-readable medium 16 may be understood toinclude one or more computer-readable media of various forms, in variousexamples.

Input interface 26 of destination device 14 receives information fromcomputer-readable medium 16. The information of computer-readable medium16 may include syntax information defined by video encoder 22 of videoencoder 22, which is also used by video decoder 30, that includes syntaxelements that describe characteristics and/or processing of blocks andother coded units, e.g., groups of pictures (GOPs). Storage media 28 maystore encoded video data received by input interface 26. Display device32 displays the decoded video data to a user, and may comprise any of avariety of display devices such as a cathode ray tube (CRT), a liquidcrystal display (LCD), a plasma display, an organic light emitting diode(OLED) display, or another type of display device.

Video encoder 22 and video decoder unit 30 each may be implemented asany of a variety of suitable encoder or decoder circuitry, such as oneor more microprocessors, processing circuitry (e.g., fixed functioncircuitry, programmable processing circuitry, or any combination offixed function circuitry and programmable processing circuitry), digitalsignal processors (DSPs), application specific integrated circuits(ASICs), field programmable gate arrays (FPGAs), discrete logic,software, hardware, firmware or any combinations thereof. When thetechniques are implemented partially in software, a device may storeinstructions for the software in a suitable, non-transitorycomputer-readable medium and execute the instructions in hardware usingone or more processors to perform the techniques of this disclosure.Each of video encoder 22 and video decoder 30 may be included in one ormore encoders or decoders, either of which may be integrated as part ofa combined encoder/decoder (CODEC) in a respective device.

In some examples, video encoder 22 and video decoder 30 may operateaccording to a video coding standard. Example video coding standardsinclude, but are not limited to, ITU-T H.261, ISO/IEC MPEG-1 Visual,ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visualand ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including itsScalable Video Coding (SVC) and Multi-View Video Coding (MVC)extensions. In addition, a new video coding standard, namely HighEfficiency Video Coding (HEVC) or ITU-T H.265, including its range andscreen content coding extensions, 3D video coding (3D-HEVC) andmultiview extensions (MV-HEVC) and scalable extension (SHVC), hasrecently been developed by the Joint Collaboration Team on Video Coding(JCT-VC) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC MotionPicture Experts Group (MPEG). Next generation video coding standards arebeing developed at the time of this disclosure, subsequently to thefinalization of ITU-H.265. Techniques of this disclosure may be used inconjunction with various video coding techniques that use or areeligible to use sign prediction, such as techniques described in thevarious standards listed herein, next-generation video coding standardsthat are currently under development, video coding techniques of JEM,etc.

In HEVC and other video coding specifications, a video sequencetypically includes a series of pictures. Pictures may also be referredto as “frames.” A picture may include three sample arrays, denotedS_(L), S_(Cb), and S_(Cr). S_(L) is a two-dimensional array (i.e., ablock) of luma samples. S_(Cb) is a two-dimensional array of Cbchrominance samples. S_(Cr) is a two-dimensional array of Cr chrominancesamples. Chrominance samples may also be referred to herein as “chroma”samples. In other instances, a picture may be monochrome and may onlyinclude an array of luma samples.

To generate an encoded representation of a picture, video encoder 22 maygenerate a set of coding tree units (CTUs). Each of the CTUs maycomprise a coding tree block of luma samples, two corresponding codingtree blocks of chroma samples, and syntax structures used to code thesamples of the coding tree blocks. In monochrome pictures or pictureshaving three separate color planes, a CTU may comprise a single codingtree block and syntax structures used to code the samples of the codingtree block. A coding tree block may be an N×N block of samples. A CTUmay also be referred to as a “tree block” or a “largest coding unit”(LCU). The CTUs of HEVC may be broadly analogous to the macroblocks ofother standards, such as H.264/AVC. However, a CTU is not necessarilylimited to a particular size and may include one or more coding units(CUs). A slice may include an integer number of CTUs orderedconsecutively in a raster scan order.

This disclosure may use the term “video unit” or “video block” or“block” to refer to one or more sample blocks and syntax structures usedto code samples of the one or more blocks of samples. Example types ofvideo units may include CTUs, CUs, PUs, transform units (TUs),macroblocks, macroblock partitions, and so on. In some contexts,discussion of PUs may be interchanged with discussion of macroblocks ormacroblock partitions. Example types of video blocks may include codingtree blocks, coding blocks, and other types of blocks of video data.

To generate a coded CTU, video encoder 22 may recursively performquad-tree partitioning on the coding tree blocks of a CTU to divide thecoding tree blocks into coding blocks, hence the name “coding treeunits.” A coding block is an N×N block of samples. A CU may comprise acoding block of luma samples and two corresponding coding blocks ofchroma samples of a picture that has a luma sample array, a Cb samplearray, and a Cr sample array, and syntax structures used to code thesamples of the coding blocks. In monochrome pictures or pictures havingthree separate color planes, a CU may comprise a single coding block andsyntax structures used to code the samples of the coding block.

Video encoder 22 may partition a coding block of a CU into one or moreprediction blocks. A prediction block is a rectangular (i.e., square ornon-square) block of samples on which the same prediction is applied. Aprediction unit (PU) of a CU may comprise a prediction block of lumasamples, two corresponding prediction blocks of chroma samples, andsyntax structures used to predict the prediction blocks. In monochromepictures or pictures having three separate color planes, a PU maycomprise a single prediction block and syntax structures used to predictthe prediction block. Video encoder 22 may generate predictive blocks(e.g., luma, Cb, and Cr predictive blocks) for prediction blocks (e.g.,luma, Cb, and Cr prediction blocks) of each PU of the CU.

Video encoder 22 may use intra prediction or inter prediction togenerate the predictive blocks for a PU. If video encoder 22 uses intraprediction to generate the predictive blocks of a PU, video encoder 22may generate the predictive blocks of the PU based on decoded samples ofthe picture that includes the PU.

After video encoder 22 generates predictive blocks (e.g., luma, Cb, andCr predictive blocks) for one or more PUs of a CU, video encoder 22 maygenerate one or more residual blocks for the CU. As one example, videoencoder 22 may generate a luma residual block for the CU. Each sample inthe CU's luma residual block indicates a difference between a lumasample in one of the CU's predictive luma blocks and a correspondingsample in the CU's original luma coding block. In addition, videoencoder 22 may generate a Cb residual block for the CU. In one exampleof chroma prediction, each sample in the Cb residual block of a CU mayindicate a difference between a Cb sample in one of the CU's predictiveCb blocks and a corresponding sample in the CU's original Cb codingblock. Video encoder 22 may also generate a Cr residual block for theCU. Each sample in the CU's Cr residual block may indicate a differencebetween a Cr sample in one of the CU's predictive Cr blocks and acorresponding sample in the CU's original Cr coding block. However, itshould be understood that other techniques for chroma prediction may beused.

Furthermore, video encoder 22 may use quad-tree partitioning todecompose the residual blocks (e.g., the luma, Cb, and Cr residualblocks) of a CU into one or more transform blocks (e.g., luma, Cb, andCr transform blocks). A transform block is a rectangular (e.g., squareor non-square) block of samples on which the same transform is applied.A transform unit (TU) of a CU may comprise a transform block of lumasamples, two corresponding transform blocks of chroma samples, andsyntax structures used to transform the transform block samples. Thus,each TU of a CU may have a luma transform block, a Cb transform block,and a Cr transform block. The luma transform block of the TU may be asub-block of the CU's luma residual block. The Cb transform block may bea sub-block of the CU's Cb residual block. The Cr transform block may bea sub-block of the CU's Cr residual block. In monochrome pictures orpictures having three separate color planes, a TU may comprise a singletransform block and syntax structures used to transform the samples ofthe transform block.

Video encoder 22 may apply one or more transforms a transform block of aTU to generate a coefficient block for the TU. For instance, videoencoder 22 may apply one or more transforms to a luma transform block ofa TU to generate a luma coefficient block for the TU. A coefficientblock may be a two-dimensional array of transform coefficients. Atransform coefficient may be a scalar quantity. Video encoder 22 mayapply one or more transforms to a Cb transform block of a TU to generatea Cb coefficient block for the TU. Video encoder 22 may apply one ormore transforms to a Cr transform block of a TU to generate a Crcoefficient block for the TU.

After generating a coefficient block (e.g., a luma coefficient block, aCb coefficient block or a Cr coefficient block), video encoder 22 mayquantize the coefficient block. Quantization generally refers to aprocess in which transform coefficients are quantized to possibly reducethe amount of data used to represent the transform coefficients,providing further compression. After video encoder 22 quantizes acoefficient block, video encoder 22 may entropy encode syntax elementsindicating the quantized transform coefficients. For example, videoencoder 22 may perform Context-Adaptive Binary Arithmetic Coding (CABAC)on the syntax elements indicating the quantized transform coefficients.

Video encoder 22 may output a bitstream that includes a sequence of bitsthat forms a representation of coded pictures and associated data. Thus,the bitstream comprises an encoded representation of video data. Thebitstream may comprise a sequence of network abstraction layer (NAL)units. A NAL unit is a syntax structure containing an indication of thetype of data in the NAL unit and bytes containing that data in the formof a raw byte sequence payload (RBSP) interspersed as necessary withemulation prevention bits. Each of the NAL units may include a NAL unitheader and encapsulates a RBSP. The NAL unit header may include a syntaxelement indicating a NAL unit type code. The NAL unit type codespecified by the NAL unit header of a NAL unit indicates the type of theNAL unit. A RBSP may be a syntax structure containing an integer numberof bytes that is encapsulated within a NAL unit. In some instances, anRB SP includes zero bits.

Video decoder 30 may receive a bitstream generated by video encoder 22.In addition, video decoder 30 may parse the bitstream to obtain syntaxelements from the bitstream. Video decoder 30 may reconstruct thepictures of the video data based at least in part on the syntax elementsobtained from the bitstream. The process to reconstruct the video datamay be generally reciprocal to the process performed by video encoder22. For instance, video decoder 30 may use motion vectors of PUs todetermine predictive blocks for the PUs of a current CU. In addition,video decoder 30 may inverse quantize coefficient blocks of TUs of thecurrent CU. Video decoder 30 may perform inverse transforms on thecoefficient blocks to reconstruct transform blocks of the TUs of thecurrent CU. Video decoder 30 may reconstruct the coding blocks of thecurrent CU by adding the samples of the predictive blocks for PUs of thecurrent CU to corresponding samples of the transform blocks of the TUsof the current CU. By reconstructing the coding blocks for each CU of apicture, video decoder 30 may reconstruct the picture.

Aspects of transform coefficient coding in HEVC and existing techniquesof coefficient sign bit prediction are discussed below. According toHEVC transform coefficient coding, the transform coefficient block (TB)is first divided into coefficient groups (CG), where each CG representsa sub-block, such as a 4×4 sub-block. For example, a 32×32 TU has atotal of 64 CGs, and a 16×16 TU has a total of 16 CGs. The entropycoding of the TB is performed in units of CGs. The CGs inside a TB arecoded according to a given scan order. When coding each CG, thecoefficients inside the current CG are scanned and coded according to acertain pre-defined scan order for 4×4 block. In JEM, the CG size couldbe either 4×4 or 2×2, depending on whether the height or width of one TBis equal to 2.

FIG. 2 is a conceptual diagram illustrating an example of a coefficientscan based on coefficient groups. More specifically, FIG. 2 illustratesthe coefficient scan for an 8×8 TB 42 containing 4 CGs in HEVC. For eachcolor component, video encoder 22 may first signal a one-bit flag tovideo decoder 30 to indicate whether a current TB (TB 42 in this case)has at least one non-zero coefficient. If there is at least one non-zerocoefficient in the current TB (TB 42), video encoder 22 may thenexplicitly encode the position of the last significant coefficient inthe coefficient scan order in the TB with a coordinate (or coordinates)relative to the top-left corner of the TB. The vertical or horizontalcomponent of the coordinate(s) is represented by its prefix and suffix,where the prefix is binarized with truncated rice (TR), and the suffixis binarized with fixed length.

With such a position encoded and signaled, and also the coefficientscanning order of the CGs, video encoder 22 may further signal a one-bitflag for all CGs except the last CG (in scanning order), which indicateswhether the respective CG contains any non-zero coefficients. For thoseCGs that contain non-zero coefficients, video encoder 22 may furtherencode and signal significant flags, absolute values of coefficients,and sign information of the non-zero coefficient(s) for each coefficientaccording to the pre-defined 4×4 coefficient scan order. In the HEVCtransform coefficient entropy coding scheme, a sign bit, if coded, isalways bypass coded. For instance, video encoder 22 and video decoder 30may not apply a context, and may always encode/decode one (1) bit foreach sign bit using an equal probability (EP) assumption.

Aspects of sign data hiding are discussed in this paragraph. For eachCG, and depending on one or more criteria, video encoder 22, when usingsign data hiding (SDH), may omit encoding the sign of the last nonzerocoefficient (in reverse scan order) which is the first nonzerocoefficient in the forward scan order. Instead, video encoder 22 mayembed the sign value in the parity of the sum of the levels of the CG,using a predefined convention. The predefined convention is described asfollows: even corresponds to “+” and odd to “−.” An example criterion touse SDH is the distance in scan order between the first and the lastnonzero coefficients of the CG. If the between the first and lastnonzero coefficients in the CG is equal or larger than four (4), thenvideo encoder 22 may use SDH. The value of four (4) was selected becauseit provides the largest gain on HEVC test sequences.

Aspects of coefficient sign prediction are described in the followingparagraphs. To improve the coding efficiency for sign bit information,coefficient sign prediction methods have been proposed in variousliterature. For instance, “Residual Coefficient Sign Prediction” byFelix Henry and Gordon Clare, Joint Video Exploration Team (WET) ofITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Doc. WET-D0031, October2016 (hereinafter, “Henry”) proposed a sign prediction method on top ofJEM.

To predict the sign for one coefficient, video decoder 30 mayreconstruct TB 42 using both positive value and negative values for therespective sign. Each block reconstruction using a candidate sign valueis referred to as a “hypothesis reconstruction.” Video decoder 30 mayevaluate the two hypothesis reconstructions using a given spatial-domaincost function, and the hypothesis which minimizes the cost functionproduces the predicted sign value. That is, video decoder 30 maygenerate a respective hypothesis reconstruction for the block, using adifferent sign or a different sign combination to generate eachrespective hypothesis reconstruction for the block.

Furthermore, to predict multiples signs for TB 42, e.g., ‘N’ number ofsigns, video decoder 30 may reconstruct TB 42 using differentcombinations of candidate sign prediction values, which includes a totalof 2N different hypothesis reconstructions. Similarly, video decoder 30may evaluate each hypothesis using a given spatial-domain cost function,and the hypothesis which minimize the cost function gives the predictedsign value combination.

The cost function typically measures spatial discontinuity betweenpreviously-reconstructed neighbor pixels and the currently-testedreconstructed block using one of the hypotheses. The hypothesis whichshows the most smooth pixel value transition at the block boundary ofthe current block is considered to be the best prediction, with respectto the hypothesis prediction techniques described above. For example, inHenry, the cost is measured using the leftmost and topmost pixels of ahypothesis reconstruction.

FIGS. 3A and 3B are diagrams illustrating hypothesis reconstruction andprediction techniques. FIG. 3A illustrates the hypothesis reconstructionand prediction techniques of Henry, which video decoder 30 use inconjunction with the techniques of this disclosure. FIG. 3A illustratesthe coordinates of various pixels of TB 42, and indicates thepreviously-reconstructed neighbor pixels as well as for the hypothesisreconstruction. FIG. 3B sets forth mathematical formulae illustratinghypothesis reconstruction and prediction techniques that video decoder30 may implement in conjunction with the techniques of this disclosure.

According to a specific sign prediction scheme described in Henry, videodecoder 30 may initially dequantize the TU, and then choose ‘n’coefficients for which sign prediction is to be performed. Video decoder30 may scan the coefficients in raster-scan order. Dequantized valuesover a predefined threshold are preferred over values lower than thepredefined threshold when collecting the ‘n’ coefficients to treat. Withthese ‘n’ values, video decoder 30 may perform simplified borderreconstructions on ‘2n’ occasions as described below, with onereconstruction per unique combination of signs for the ‘n’ coefficients.

To reduce the complexity of performing sign prediction, video decoder 30may perform a template-based hypothesis reconstruction. For a particularhypothesis reconstruction, video decoder 30 may recreate only theleftmost and topmost pixels of the block, from the inversetransformation added to the block prediction. Although (according to theuse-case scenario illustrated in FIG. 3A) the first (vertical) inversetransform is complete, the second (horizontal) inverse transform onlyhas to create the leftmost and topmost pixel outputs, and is thusfaster. An additional flag, “topLeft”, has been added to inversetransform functions to allow this.

In addition, the number of inverse transform operations performed isreduced by using a system of ‘templates’. In this way, when predicting‘n’ signs in a block, video decoder 30 may only perform ‘n+1’ number ofinverse transform operations. Examples of the steps performed are listedbelow:

1. Video decoder 30 may perform a single inverse transform operating onthe dequantized coefficients, where the values of all signs beingpredicted are set to be positive. Once added to the prediction of thecurrent block, this corresponds to the border reconstruction for thefirst hypothesis.2. For each of the ‘n’ coefficients for which video decoder 30 predictssigns, an inverse transform operation is performed on an otherwise emptyblock containing the corresponding dequantized (and positive)coefficient as its only non-null element. The leftmost and topmostborder values are saved in what is termed a ‘template’ for use duringlater reconstructions.

Video decoder 30 may start border reconstruction for a later hypothesisby taking an appropriate saved reconstruction of a previous hypothesiswhich only uses a single predicted sign to be changed from positive tonegative in order to construct the desired current hypothesis. Videodecoder 30 may then approximate this change of sign by the doubling andsubtraction from the hypothesis border of the template corresponding tothe sign being predicted. The border reconstruction, after applying costfunction measurement, is then saved, if the border reconstruction isavailable to video decoder 30 to be reused for constructing one or morelater hypotheses.

Table 1 below maps a template name to a corresponding template

TABLE 1 Template Name How to Create T001 inv xform single +ve 1^(st)sign-hidden coeff T010 inv xform single +ve 2^(nd) sign-hidden coeffT100 inv xform single +ve 3^(rd) sign-hidden coeff

Table 2 below shows save/restore and template application for a 3 sign,8 entry use case

TABLE 2 Hypothesis How to Create Store for later reuse as H000 inv xformall coeffs H000 add to pred H001 H000 − 2*T001 H010 H000 − 2*T010 H010H011 H010 − 2*T001 H100 H000 − 2*T100 H100 H101 H100 − 2*T001 H110 H100− 2*T010 H110 H111 H110 − 2*T001

In some examples, video decoder 30 may use these approximations onlyduring the process of sign prediction, and not during finalreconstruction. For a transform coefficient with larger magnitude, thesign prediction generally yields a better chance to achieve a correctprediction. This is because an incorrect sign prediction for a transformcoefficient with larger magnitude typically shows more discrepancy withrespect to boundary sample smoothness.

With sign prediction, instead of coding the explicit sign value, videoencoder 22 and video decoder 30 may encode/decode the correctness of thesign prediction. For example, for predicting a coefficient sign whichactually has a positive value, if the predicted sign is also positive(e.g. if the sign prediction is correct), video encoder 22 may encode,and video decoder 30 may decode, a ‘0’ bin. Otherwise, if the predictedsign is negative, (e.g., if the sign prediction is not correct), videoencoder 22 may encode, and video decoder 30 may decode, a ‘1’ bin. Inthis way, sign prediction may utilize the level value (e.g., magnitude)of the transform coefficient as the context for coding the correctnessof sign prediction, because a larger magnitude of transform coefficienttends (or ‘leans’) towards a higher chance of a ‘0’ bin.

Existing sign prediction techniques may cause one or more potentialissues. As one example, the existing sign prediction technology showsrelatively high complexity for both encoders and decoders, because theexisting sign prediction techniques may use multiple inverse transformprocesses to generate the leftmost and topmost pixel samples during acoding run, or “on the fly.” For instance, four (4) signs would resultin a total of sixteen (16) possibilities, with multiple reconstructionsfor each hypothesis. As a result, video decoder 30 would perform inversetransform, inverse quantization, and reconstruction sixteen (16) timesin each case. The increased complexity may be undesirable for apractical video codec. Moreover, for larger numbers, the number ofpredicted signs may increase at a fast rate. That is, the complexityincrease of existing sign prediction technology may increase quickly ifa relatively larger number of signs are predicted. In some examples,there may be 6, 7, 8, or a greater number of predicted signs for a givenblock.

Various techniques of this disclosure are generally directed tomitigating or potentially eliminating one or more of the issuesdiscussed above, with respect to sign prediction. Aspects of the varioustechniques of this disclosure are described as being performed by videoencoder 22 and video decoder 30 below. It will be appreciated, however,that the techniques described herein may be performed by various videocoding devices, including video encoders that are configured to decodeencoded video data (e.g., via a “decoding loop”).

FIG. 4 is a block diagram illustrating an example video encoder 22 thatmay implement the techniques of this disclosure. FIG. 4 is provided forpurposes of explanation and should not be considered limiting of thetechniques as broadly exemplified and described in this disclosure. Thetechniques of this disclosure may be applicable to various codingstandards or methods.

In the example of FIG. 4, video encoder 22 includes a predictionprocessing unit 100, video data memory 101, a residual generation unit102, a transform processing unit 104, a quantization unit 106, aninverse quantization unit 108, an inverse transform processing unit 110,a reconstruction unit 112, a filter unit 114, a decoded picture buffer116, and an entropy encoding unit 118. Prediction processing unit 100includes an inter-prediction processing unit 120 and an intra-predictionprocessing unit 126. Inter-prediction processing unit 120 may include amotion estimation unit and a motion compensation unit (not shown).

Video data memory 101 may be configured to store video data to beencoded by the components of video encoder 22. The video data stored invideo data memory 101 may be obtained, for example, from video source18. Decoded picture buffer 116 may be a reference picture memory thatstores reference video data for use in encoding video data by videoencoder 22, e.g., in intra- or inter-coding modes. Video data memory 101and decoded picture buffer 116 may be formed by any of a variety ofmemory devices, such as dynamic random-access memory (DRAM), includingsynchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM(RRAM), or other types of memory devices. Video data memory 101 anddecoded picture buffer 116 may be provided by the same memory device orseparate memory devices. In various examples, video data memory 101 maybe on-chip with other components of video encoder 22, or off-chiprelative to those components. Video data memory 101 may be the same asor part of storage media 20 of FIG. 1.

Video encoder 22 receives video data. Video encoder 22 may encode eachCTU in a slice of a picture of the video data. Each of the CTUs may beassociated with equally-sized luma coding tree blocks (CTBs) andcorresponding CTBs of the picture. As part of encoding a CTU, predictionprocessing unit 100 may perform partitioning to divide the CTBs of theCTU into progressively-smaller blocks. The smaller blocks may be codingblocks of CUs. For example, prediction processing unit 100 may partitiona CTB associated with a CTU according to a tree structure. In accordancewith one or more techniques of this disclosure, for each respectivenon-leaf node of the tree structure at each depth level of the treestructure, there are a plurality of allowed splitting patterns for therespective non-leaf node and the video block corresponding to therespective non-leaf node is partitioned into video blocks correspondingto the child nodes of the respective non-leaf node according to one ofthe plurality of allowable splitting patterns.

Video encoder 22 may encode CUs of a CTU to generate encodedrepresentations of the CUs (i.e., coded CUs). As part of encoding a CU,prediction processing unit 100 may partition the coding blocksassociated with the CU among one or more PUs of the CU. Thus, each PUmay be associated with a luma prediction block and corresponding chromaprediction blocks. Video encoder 22 and video decoder 30 may support PUshaving various sizes. As indicated above, the size of a CU may refer tothe size of the luma coding block of the CU and the size of a PU mayrefer to the size of a luma prediction block of the PU. Assuming thatthe size of a particular CU is 2N×2N, video encoder 22 and video decoder30 may support PU sizes of 2N×2N or N×N for intra prediction, andsymmetric PU sizes of 2N×2N, 2N×N, N×2N, N×N, or similar for interprediction. Video encoder 22 and video decoder 30 may also supportasymmetric partitioning for PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2Nfor inter prediction.

Inter-prediction processing unit 120 may generate predictive data for aPU by performing inter prediction on each PU of a CU. The predictivedata for the PU may include predictive blocks of the PU and motioninformation for the PU. Inter-prediction processing unit 120 may performdifferent operations for a PU of a CU depending on whether the PU is inan I slice, a P slice, or a B slice. In an I slice, all PUs are intrapredicted. Hence, if the PU is in an I slice, inter-predictionprocessing unit 120 does not perform inter prediction on the PU. Thus,for blocks encoded in I-mode, the predicted block is formed usingspatial prediction from previously-encoded neighboring blocks within thesame frame. If a PU is in a P slice, inter-prediction processing unit120 may use uni-directional inter prediction to generate a predictiveblock of the PU. If a PU is in a B slice, inter-prediction processingunit 120 may use uni-directional or bi-directional inter prediction togenerate a predictive block of the PU.

Intra-prediction processing unit 126 may generate predictive data for aPU by performing intra prediction on the PU. The predictive data for thePU may include predictive blocks of the PU and various syntax elements.Intra-prediction processing unit 126 may perform intra prediction on PUsin I slices, P slices, and B slices.

To perform intra prediction on a PU, intra-prediction processing unit126 may use multiple intra prediction modes to generate multiple sets ofpredictive data for the PU. Intra-prediction processing unit 126 may usesamples from sample blocks of neighboring PUs to generate a predictiveblock for a PU. The neighboring PUs may be above, above and to theright, above and to the left, or to the left of the PU, assuming aleft-to-right, top-to-bottom encoding order for PUs, CUs, and CTUs.Intra-prediction processing unit 126 may use various numbers of intraprediction modes, e.g., 33 directional intra prediction modes. In someexamples, the number of intra prediction modes may depend on the size ofthe region associated with the PU.

Prediction processing unit 100 may select the predictive data for PUs ofa CU from among the predictive data generated by inter-predictionprocessing unit 120 for the PUs or the predictive data generated byintra-prediction processing unit 126 for the PUs. In some examples,prediction processing unit 100 selects the predictive data for the PUsof the CU based on rate/distortion metrics of the sets of predictivedata. The predictive blocks of the selected predictive data may bereferred to herein as the selected predictive blocks.

Residual generation unit 102 may generate, based on the coding blocks(e.g., luma, Cb and Cr coding blocks) for a CU and the selectedpredictive blocks (e.g., predictive luma, Cb and Cr blocks) for the PUsof the CU, residual blocks (e.g., luma, Cb and Cr residual blocks) forthe CU. For instance, residual generation unit 102 may generate theresidual blocks of the CU such that each sample in the residual blockshas a value equal to a difference between a sample in a coding block ofthe CU and a corresponding sample in a corresponding selected predictiveblock of a PU of the CU.

Transform processing unit 104 may perform quad-tree partitioning topartition the residual blocks associated with a CU into transform blocksassociated with TUs of the CU. Thus, a TU may be associated with a lumatransform block and two chroma transform blocks. The sizes and positionsof the luma and chroma transform blocks of TUs of a CU may or may not bebased on the sizes and positions of prediction blocks of the PUs of theCU. A quad-tree structure known as a “residual quad-tree” (RQT) mayinclude nodes associated with each of the regions. The TUs of a CU maycorrespond to leaf nodes of the RQT.

Transform processing unit 104 may generate transform coefficient blocksfor each TU of a CU by applying one or more transforms to the transformblocks of the TU. Transform processing unit 104 may apply varioustransforms to a transform block associated with a TU. For example,transform processing unit 104 may apply a discrete cosine transform(DCT), a directional transform, or a conceptually similar transform to atransform block. In some examples, transform processing unit 104 doesnot apply transforms to a transform block. In such examples, thetransform block may be treated as a transform coefficient block.

Quantization unit 106 may quantize the transform coefficients in acoefficient block. The quantization process may reduce the bit depthassociated with some or all of the transform coefficients. For example,an n-bit transform coefficient may be rounded down to an m-bit transformcoefficient during quantization, where n is greater than m. Quantizationunit 106 may quantize a coefficient block associated with a TU of a CUbased on a quantization parameter (QP) value associated with the CU.Video encoder 22 may adjust the degree of quantization applied to thecoefficient blocks associated with a CU by adjusting the QP valueassociated with the CU. Quantization may introduce loss of information.Thus, quantized transform coefficients may have lower precision than theoriginal ones.

Inverse quantization unit 108 and inverse transform processing unit 110may apply inverse quantization and inverse transforms to a coefficientblock, respectively, to reconstruct a residual block from thecoefficient block. Reconstruction unit 112 may add the reconstructedresidual block to corresponding samples from one or more predictiveblocks generated by prediction processing unit 100 to produce areconstructed transform block associated with a TU. By reconstructingtransform blocks for each TU of a CU in this way, video encoder 22 mayreconstruct the coding blocks of the CU.

Filter unit 114 may perform one or more deblocking operations to reduceblocking artifacts in the coding blocks associated with a CU. Decodedpicture buffer 116 may store the reconstructed coding blocks afterfilter unit 114 performs the one or more deblocking operations on thereconstructed coding blocks. Inter-prediction processing unit 120 mayuse a reference picture that contains the reconstructed coding blocks toperform inter prediction on PUs of other pictures. In addition,intra-prediction processing unit 126 may use reconstructed coding blocksin decoded picture buffer 116 to perform intra prediction on other PUsin the same picture as the CU.

Entropy encoding unit 118 may receive data from other functionalcomponents of video encoder 22. In various examples, entropy encodingunit 118 may be configured to implement one or more techniques of thisdisclosure. For example, entropy encoding unit 118 may receivecoefficient blocks from quantization unit 106 and may receive syntaxelements from prediction processing unit 100. Entropy encoding unit 118may perform one or more entropy encoding operations on the data togenerate entropy-encoded data. For example, entropy encoding unit 118may perform a CABAC operation, a context-adaptive variable length coding(CAVLC) operation, a variable-to-variable (V2V) length coding operation,a syntax-based context-adaptive binary arithmetic coding (SBAC)operation, a Probability Interval Partitioning Entropy (PIPE) codingoperation, an Exponential-Golomb encoding operation, or another type ofentropy encoding operation on the data. Video encoder 22 may output abitstream that includes entropy-encoded data generated by entropyencoding unit 118. For instance, the bitstream may include data thatrepresents a RQT for a CU.

According to some aspects of this disclosure, video encoder 22 may use apre-stored set of reconstructed residual samples to perform hypothesisreconstruction. Techniques of this disclosure by which video encoder 22uses pre-stored reconstructed residual samples are generally referred toherein as “template-based reconstruction” techniques. For instance, thepre-stored reconstructed residual samples may be stored to any memoryaccessible (e.g., via direct or indirect communicative coupling) toprocessing circuitry of video encoder 22, such as storage media 20 ofsource device 12 and/or video data memory 101. In some instances oftemplate-based reconstruction according to aspects of this disclosure,video encoder 22 may store one or more fully-reconstructed and/orpartially-reconstructed residual blocks, and then use some combination(e.g., a linear combination) of the stored information to form atemplate. In turn, video encoder 22 may use the template to derive allof the possible reconstructed values. Each stored set of residual blocksmay be pre-calculated using a pre-defined transform coefficient block.To perform hypothesis reconstruction, video encoder 22 may identify aselected group of fully- and/or partially-reconstructed residual blocksthe stored set, and generate/estimate the hypothesis reconstructionusing some combination the reconstructed residual blocks identified inthis fashion.

As discussed above, video encoder 22 (and/or components thereof) may beconfigured, according to aspects of this disclosure, to mitigate variousissues associated with existing sign prediction technology. According tosome examples of the template-based reconstruction techniques of thisdisclosure, video encoder 22 may form each reconstructed block of thestored set using a pre-defined quantized transform coefficient block(TB), a given dequantization parameter, and a given primary transformand/or a given secondary transform. An example is discussed below withrespect to a particular coefficient, for which the position is denotedas f(x,y) in an M×N TB, where 0<=x<M, 0<=y<N. Given the inversetransform method (including primary transform and/or secondarytransform) and dequantization parameter, video encoder 22 may generate aquantized coefficient block (denoted as B) by setting f(x,y) to acertain value, and by setting all other coefficients to a differentvalue from the value assigned to f(x,y). For example, video encoder 22may set the value of f(x,y) to one (1), and may set all othercoefficients to zero (0).

In turn, video encoder 22 may pre-calculate a fully- orpartially-reconstructed residual block (denoted as R) by applying thegiven dequantization and inverse transform on the quantized coefficientblock B. Video encoder 22 may pre-store the reconstructed residual blockR to a memory (e.g., to storage media 20 of source device 12 and/or tovideo data memory 101). By pre-storing the partially- orfully-reconstructed block R, video encoder 22 may leverage existingresources (e.g., in the form of storage media 20 and/or video datamemory 101) to cache pre-calculated information that can be used toimprove the efficiency of calculations to be performed at a later stage.In this manner, during sign prediction operations, if the sign ofcoefficient f(x,y) needs to be predicted in an M×N TB with the sameinverse transform and dequantization parameter, the techniques of thisdisclosure enable video encoder 22 to avoid performing an inversetransform for f(x,y). Instead, video encoder 22 may scale the pre-storedreconstructed residual block R by a factor of |f(x,y)| and then use thescaled version of the pre-stored reconstructed residual block R toestimate the reconstructed block.

In another example of the template-based reconstruction techniques ofthis disclosure, video encoder 22 may pre-store a dequantized version ofthe TB. That is, in these examples, video encoder 22 may reconstruct thecurrent block using dequantized versions of the (fully or partially)reconstructed blocks stored to storage media 20. In other words,according to these examples, video encoder 22 may reconstruct thecurrent block using a pre-defined dequantized TB, and one or both of agiven primary transform or a given secondary transform.

One use-case scenario of the dequantized TB-based implementations of thetemplate-based reconstruction techniques of this disclosure is describedbelow with respect to a particular coefficient f(x,y) in an M×N TB,where 0<=x<M, 0<=y<N, given the inverse transform method (includingprimary transform and/or secondary transform). In this example, videoencoder 22 may generate a dequantized coefficient block B by settingf(x,y) to a certain value, and by setting all other coefficients toanother value. For instance, video encoder 22 may set f(x,y) to a valueof one (1), and may set all of the remaining coefficients to a value ofzero (0). In turn, video encoder 22 may pre-calculate a fully orpartially reconstructed residual block R by applying the given inversetransform on B, and may pre-store the reconstructed residual block R toa memory (e.g., to storage media 20 and/or video data memory 101).

In this way, during sign prediction, if the sign of the quantizedcoefficient f(x,y) is to be predicted in an M×N TB with the same inversetransform, the techniques of this disclosure enable video encoder 22and/or video decoder 30 to avoid doing inverse transform for f(x,y).Instead, video encoder 22 may first dequantize the quantized coefficientf(x,y) to be f(x,y)′ (to denote the dequantized coefficient), then videoencoder 22 ma scale the pre-stored reconstructed residual block R by afactor of |f(x,y)′|. Video encoder 22 may then use the scaled version(i.e., scaled using the dequantized value) of the pre-stored dequantizedreconstructed residual block R to estimate the reconstructed block. Asillustrated in the discussion above, video encoder 22 may implement thetemplate-based reconstruction techniques of this disclosure byoptionally using dequantized TB information for the pre-stored template,with varying computation complexity arising from the optional use ornon-use of the dequantization operations.

In some implementations of the template-based reconstruction techniquesof this disclosure, video encoder 22 may pre-store the set of partially-and/or fully-reconstructed residual blocks with fractional precision. Insome examples, video encoder 22 may first scale the set of reconstructedresidual blocks with a pre-defined factor (e.g., 2, 4, 8, 16, etc.), andthen round the result to the nearest integer, and store the rounded,scaled result to storage media 20 and/or video data memory 101. In thisway, video encoder 22 may enhance the precision of the pre-storedreconstructed residual blocks. Moreover, in both the fractionalprecision implementations and the scaling/rounding based implementationsdescribed above, video encoder 22 may avail of the benefits of thetemplate-based reconstruction techniques of this disclosure, whilereducing the memory requirements with respect to storage media 20 and/orvideo data memory 101.

According to some implementations of the template-based reconstructiontechniques of this disclosure, video encoder 22 may pre-store the set offully- and/or partially-reconstructed residual blocks with clippedvalues. In some instances, video encoder 22 may clip the values of thereconstructed residual blocks before storage, such that the clippedvalues fit a restricted bit range of representation. For example, byclipping the values using constraints of a minimum value (Vmin) ofnegative eight (−8) and a maximum value (Vmax) of seven (7), videoencoder 22 may form the entries of the pre-stored reconstructed residualvalues such that all of the pre-stored reconstructed residual values canbe represented using, at most, four (4) bits. In various use-casescenarios, video encoder 22 may utilize one or more of the factionalprecision implementation, the scaling/rounding implementation, or theclipping-based implementations of the template-based reconstructiontechniques individually or in any combination, in accordance withvarious aspects of this disclosure.

In some examples of the template-based reconstruction techniques of thisdisclosure, video encoder 22 may use a linear combination of certainidentified reconstructed residual blocks of the stored reconstructedresidual blocks to reconstruct the current TB. For example, the linearcombination may represent a weighted sum of the identified storedreconstructed residual blocks. To calculate the weighted sum, videoencoder 22 may determine the respective weights using the magnitude(also referred to as ‘level value’) of a quantized or dequantized (asthe case may be) transform coefficient for which the sign is beingpredicted. By using a weighted sum of multiple stored reconstructedresidual blocks, video encoder 22 may improve hypothesis precision byutilizing data from multiple previously-reconstructed blocks.

Video encoder 22 may, in some examples, implement template-basedreconstruction techniques of this disclosure in a manner that makes usof offline computation. Because different transform and/or quantizationmethods may be available to video encoder 22, video encoder 22 may needto store multiples of reconstructed blocks for a single quantizedcoefficient block. To mitigate storage requirements and to ease theresource consumption of accessing the pre-calculated residual blocks,video encoder 22 may avail of offline computation capabilities. Forinstance, video encoder 22 may perform a merging process on thepre-stored residual blocks, such that multiple residual blocks aremerged and represented by one residual block. In some examples, videoencoder 22 may pre-define a function ƒ to derive one pre-stored residualblock from another pre-stored residual block, such that only one ofseveral residual blocks may need to be stored. That is, in theseexamples, video encoder 22 may store one residual block, and may deriveother residual blocks using the pre-defined function ƒ and the singlestored residual block.

In some implementations of the template-based reconstruction techniquesof this disclosure, video encoder 22 may store pre-calculated residualblocks to off-chip memory (also referred to as “external” memory) ofsource device 12. The total size of the set of pre-calculated residualblocks may, in some instances, be large, and therefore may be difficultto store in on-chip memory. To continue using the template-basedreconstruction techniques of this disclosure while operating within theavailable storage constraints, video encoder 22 may pre-store the set ofpre-calculated residual blocks to off-chip memory, which, in someexamples, tends to represent a cheaper memory than on-chip memory. Inthese implementations, video encoder 22 may only load the residualblocks, as needed, to on-chip memory. For instance, video encoder 22 mayload the residual blocks once per bigger block or bigger picture area,such as for a largest coding unit (LCU).

According to some implementations of the template-based reconstructiontechniques of this disclosure, video encoder 22 may use each of thepre-stored set of partial residual blocks as a template for signprediction. In some examples, video encoder 22 may store only theleftmost and topmost residual pixels for the pre-calculated residualblocks. In one example, video encoder 22 may store the top-left residualpixel, in addition to one or more residual pixels of the top row(starting from the residual pixel positioned immediately right of thetop-left residual pixel) and/or one or more residual pixels of theleftmost column (starting from the residual pixel positioned immediatelybelow of the top-left residual pixel). In some examples, video encoder22 may apply the template-based sign prediction process described inHenry in conjunction with the pre-stored set of partial residual blocks(of this disclosure) in such a way that the template is not generated bydoing partial inverse transform on the fly. Instead, video encoder 22may generate the template by accessing the pre-stored set of partialresidual blocks, and performing a scaling operation.

Video encoder 22 may, in some examples, be configured according toaspects of this disclosure to perform selective sign prediction. Forinstance, video encoder 22 may perform selective sign prediction withrespect to lower-frequency transform coefficients. Video encoder 22 mayalso perform selective sign prediction to avoid sign prediction foroutlier block sizes (e.g. very large or very small block sizes).Examples of outlier dimensions that video encoder 22 may use inaccordance with the techniques of this disclosure are discussed below.To perform the selective sign prediction techniques of this disclosure,video encoder 22 may determine various conditions under which signprediction is to be performed or alternatively skipped. Variousconditions for the selective sign selection aspects of this disclosureare described in the following paragraphs.

Video encoder 22 may apply several of the conditions for the selectivesign prediction aspects of this disclosure on a block-wide basis. Theseconditions may relate to various features including, but not limited to,the block width, block height, block area (also referred to as “size”),transform coefficient frequency locations, last position, intraprediction mode, whether only a primary transform is used to code ablock, whether only particular transform functions or certain transformsubset are used for coding the block, inverse transform method, whetherthe block is intra coded or inter coded blocks, or any other decodedinformation of the current and/or neighbor block(s).

According to some implementations of the selective sign predictiontechniques of this disclosure, video encoder 22 may not apply signprediction if either the width of the currently-encoded block or theheight of the currently-encoded block is larger than a pre-definedthreshold. In contrast, according to existing sign predictiontechnology, sign prediction would be applied regardless of the blockwidth or height, as the case may be. In some examples, the threshold maybe configurable, and video encoder 22 may signal the threshold. Invarious examples, video encoder 22 may set the threshold to valuesincluding, but not limited to 4, 8, 16, 32, or 64.

In some examples of the selective sign prediction techniques of thisdisclosure, video encoder 22 may configure the threshold value to bedependent on coding modes used to encode the block. For example, videoencoder 22 may configure the threshold to be different for intra-codedblocks and inter-coded blocks. In one example, video encoder 22 may setthe threshold value to 32 for intra-coded blocks, while setting thethreshold value to 64 for inter-coded block. In some examples, videoencoder 22 may not apply sign prediction if the minimum or maximum ofblock width or block height exceeds a threshold.

According to some examples of the sign prediction techniques of thisdisclosure, video encoder 22 may not apply sign prediction for transformcoefficients that are not located in the top-left K×L part of a TB. K×Lmay represent a square sub-block of the TB, such as a sub-block having adimensionality of 2×2, 3×3, 4×4 or 8×8, etc. In contrast, according toexisting sign prediction technology, sign prediction would be appliedfor transform coefficients positioned outside of the top-left K×Lportion of the TB. In some examples, the values of K and/or L mayfurther depend on coding modes and/or quantization parameters. In thespecific example of the K×L portion of the TB representing a 2×2sub-block, there may be only four (4) possible predicted signs.

In some implementations of the selective sign prediction techniques ofthis disclosure, video encoder 22 may limit the inverse transformperformed in the sign prediction process to certain inversetransform(s). For example, video encoder 22 may impose the inversetransform constraint while generating a hypothesis during the hypothesisprediction process. The selected transform(s) can be different from theinverse transform which video encoder 22 applies to generate the actualreconstructed block after deriving the actual coefficient signs.

In one such example, video encoder 22 may only use discrete cosinetransform type-II (DCT-II) as the primary inverse transform during signprediction. In another such example, during sign prediction, videoencoder 22 may replace the secondary transform with a transform using apre-defined permutation matrix. The pre-defined permutation matrix mayinclude entities of “+1” and “−1”. In another example, video encoder 22may not use a secondary transform during sign prediction.

According to some examples of the selective sign prediction techniquesof this disclosure, video encoder 22 may apply sign prediction only forthose blocks that are encoded using only a primary transform. Bylimiting sign prediction in this manner, video encoder 22 may reducecoding complexity, such that the complexity of the inverse transform maybe smaller than the complexity arising from having a primary and asecondary transform applied together. In another example, video encoder22 may apply sign prediction only for those blocks that are encodedusing a certain transform core or certain transform set/subset. Limitingsign prediction in this manner may yield potential benefits because itis possible that sign prediction may not work efficiently for alltransform basis/cores, and coding complexity can thereby be reduced bylimiting the application of sign prediction in this way.

Some aspects of this disclosure are directed to techniques for costfunction measurement. According to the cost function measurement aspectsof this disclosure, video encoder 22 may utilize previously-decodedinformation (obtained via the reconstruction loop) to measure the costfunction for each hypothesis reconstruction. The previously-decodedinformation that video encoder 22 may leverage for cost functionmeasurement includes, but is not limited to, the width of the currentblock, the height of the current block, the current block's area (alsoreferred to as “size” herein), transform coefficient frequencylocations, a last position, an intra prediction mode used for encodingthe block, whether the evaluated block(s) are intra-coded orinter-coded, or any encoded information of the current and/or neighborblock(s).

In some examples, video encoder 22 (or one or more components thereof)may implement the cost function measurement aspects of this disclosureby measuring a difference between reconstructed samples of theneighboring block(s) and the reconstructed samples of the current blocklocated along the boundary. The measured difference is also referred toherein as a “gradient” to illustrate that a difference can be expressedas the slope at a given point of a curve. That is, instead of measuringthe first order information representing the sample values, videoencoder 22 may implement the cost function measurement techniques ofthis disclosure to measure second order (difference or gradient)information that is comparative between the neighboring reconstructedsamples and the reconstructed samples of the current block positionedalong the boundary.

According to this implementation, video encoder 22 may generateestimated boundary residual samples by subtracting the neighboringreconstructed samples from the prediction samples located at the currentblock boundary (or vice versa). That is, video encoder 22 may use thesubtraction operation(s) to obtain the second order informationrepresented by the difference between the estimated boundary residualsamples and the candidate boundary residual samples. For each candidatecombination of signs, the generated residual samples represent a set of“candidate residual samples.” By using the second order gradientinformation between the neighboring residual samples and theboundary-interfacing residual samples of the current block, videoencoder 22 may avoid instances of duplicate reconstruction processes, inwhich the same prediction samples are used to produce candidate residualsamples multiple times. In this way, techniques of this disclosureenable video encoder 22 to reduce computing resource consumption andconserve bandwidth that would otherwise be expended for redundantcandidate residual sample generation and signaling of the redundantcandidate residual sample information.

In some examples of the cost function measurement techniques of thisdisclosure, video encoder 22 may be configured to measure the pixelvalue difference (also referred to herein as a “pixel gradient value”)along an intra-prediction direction to measure the cost function. In onesuch example, video encoder 22 may pre-select a partial subset ofallowed intra prediction directions, and may map each allowed intraprediction direction to one of the pre-selected intra directions whichcan be used to approximate the intra prediction direction. In thisexample, video encoder 22 may measure the pixel value difference alongthe mapped direction in the cost function calculation.

In cases where the intra prediction direction is not available to videoencoder 22 (e.g., if one of the evaluated blocks is inter-coded), videoencoder 22 may instead derive an intra prediction mode. Video encoder 22may derive the intra prediction mode from the spatial neighboring codedblocks, or a block identified by a motion vector in a coded picture. Ifthe intra prediction direction points to a fractional position, videoencoder 22 may apply an interpolation process to derive the value of thefractional position. Video encoder 22 may apply the interpolation filterused for generating the intra-predicted block, or may use a different(e.g., simpler) interpolation filter instead.

According to some of the cost function measurement aspects of thisdisclosure, video encoder 22 may measure the cost function using asecond or higher order (N-th order, where ‘N’ represents an integervalue of two or greater) order of the pixel value difference. Forinstance, video encoder 22 may use multiple reference lines (e.g., twoor more) to measure the cost value. The number of reference lines thatvideo encoder 22 uses for measuring the cost value may depend on codedinformation, including, but not limited to, the intra prediction modeand/or neighboring reconstructed values. Video encoder 22 may, in someinstances, set the maximum number of reference lines to be equal to therequired number of reference lines for intra prediction.

According to some implementations of the cost function measurementaspects of this disclosure, video encoder 22 may exclude some pixelsfrom the cost function measurement operations, depending on whether oneor more given criteria are satisfied. In one example, video encoder 22uses a criterion of whether the absolute value of a first order, secondorder (gradient), or N-th order derivative is less than a giventhreshold value. In another example, video encoder 22 uses a criterionof whether the neighboring reconstructed samples used to calculate theN-th order derivative is available. In another example still, videoencoder 22 uses a criterion of whether identity transform or transformskip is applied to the horizontal or vertical transform.

According to some implementations of the cost function measurementaspects of this disclosure, video encoder 22 may, when measuring thecost using neighboring reconstructed values, calculate the cost valuesfor multiple directions. That is, video encoder 22 may calculate thecost values for multiple directions instead of calculating the firstorder, second order (gradient), or N-th order differences along a fixedor pre-defined direction (e.g., horizontal direction for a left blockboundary and vertical direction for a top block boundary). Rather,according to these implementations, video encoder 22 calculates the costvalues for multiple directions, and uses the minimum cost for measuringthe cost.

In some examples of the cost function measurement techniques of thisdisclosure, for relatively larger block sizes, video encoder 22 maycalculate the cost function using a downsampled hypothesisreconstruction. By using a downsampled hypothesis reconstruction forcost function measurement, video encoder 22 may reduce the complexity ofcost function application with respect to larger block sizes. Accordingto some of the cost function measurement techniques of this disclosure,video encoder 22 may measure the pixel value difference only for partialpixels in the reconstructed block and neighbor block(s). Partial pixelsmay be, for example, corner pixels. Errors typically tend to be largerfor corner pixels. In another example, video encoder 22 may exclude thecorner pixels from the difference measure based on the corner pixelsbeing potential outliers, or may assign different weights to theselected pixels.

According to some implementations of the cost function measurementaspects of this disclosure, video encoder 22 may determine the number ofpredicted signs (e.g., 0, 1, 2, 3, 4, 5, 6, etc.) in a way that thenumber of predicted signs depends on previously-encoded information.Non-limiting examples of previously-encoded information that videoencoder 22 may use in determining the number of predicted signs include,but are not limited to, intra prediction mode, block width/height, blocksize (area), QP, color components, temporal layer, whether the block isintra-coded or inter coded, whether transform skip or identity transformis applied, whether non-separable secondary transform is applied,quantized coefficients, the presence/value of an enhanced multipletransform (EMT) index, or the presence/value of a position-dependentprediction combination (PDPC) flag or index.

According to some implementations of the cost function measurementaspects of this disclosure, video encoder 22 may, when multiple ‘M’signs are to be predicted, perform sign prediction in a unit or groupmanner. That is, video encoder 22 may perform sign prediction for agroup, instead of testing all the possible 2̂M hypothesisreconstructions, where the ‘M’ signs are divided into ‘N’ groups. Videoencoder 22 may predict each group of signs by setting the remaininggroups of signs as previously-predicted values or default value, if theremaining group of signs have not yet been predicted. Video encoder 22may iterate the group-based sign prediction process for multiple roundsuntil a pre-defined limitation is met. Video encoder 22 may predict agroup of signs with larger transform coefficient levels prior topredicting a group of signs with relatively lower coefficient levels.Video encoder 22 may terminate the iterations when the predicted signsfor each group no longer change, or the iteration number has met apre-defined threshold value, or the cost difference between iterations(for example, succeeding iterations) has stopped exceeding a certainthreshold.

The following paragraphs describe aspects of this disclosure thatgenerally relate to the selection of coefficient signs to be predicted.According to the coefficient sign selection aspects of this disclosure,video encoder 22 may select and/or determine predicted transformcoefficients by utilizing already-available decoded information (e.g.,via a decoding loop implemented by video encoder 22). Thepreviously-decoded information that video encoder 22 may use forcoefficient sign selection includes, but is not limited to, one or moreof the width of the current block, the height of the current block, thearea (also referred to as “size”) of the current block, transformcoefficient frequency locations within the current block, last position,an intra prediction mode used to encode the current and/or neighboringblock(s), whether sign data hiding is applied with respect to thecurrent and/or neighboring block(s), whether the current and/orneighboring block(s) are intra-coded or inter-coded, or any encodedinformation of the current and/or neighbor block(s). In variousexamples, video encoder 22 (or one or more components thereof) mayimplement the coefficient sign selection aspects of this disclosure suchthat the maximum number of predicted signs depends on whether thecurrent block is inter-coded or intra-coded. In some examples, videoencoder 22 may set the maximum number of predicted signs based oninverse transform method (e.g., which primary and/or secondarytransform) is applied for dequantizing the block.

In some examples, video encoder 22 may apply sign hiding even when signprediction is applied the block (TB). In these examples, video encoder22 may hide the sign of the last nonzero coefficient in the forwardscanning order, instead of hiding the first nonzero coefficient in theforward scanning order. Furthermore, in some such examples, videoencoder 22 may not perform sign data hiding at the CG-level. That is, inthese instances, consecutive nonzero coefficients, even if located indifferent CGs, may be jointly considered for sign data hiding.

In some examples, video encoder 22 may pre-define several thresholdvalues (denoted by t₀, t₁, . . . , t_(N)) where t₀>t₁> . . . >t_(N). Inthese examples, video encoder 22 may derive several intervals as (t₀,+∞), [t₁, t₀), [t_(N), t_(N-1)) in descending order. Moreover, videoencoder 22 may compare dequantized coefficient levels to each one of thegiven intervals until video encoder 22 identifies or detects oneinterval which covers the dequantization coefficient level. Videoencoder 22 may identify the signs of coefficients as predicted signs,according to the descending order of the threshold values.

For instance, video encoder 22 may first place the signs forcoefficients in the interval (t₀, +∞) into the list of predicted signs,and then place the signs for coefficients in the interval [t₁, t₀) intothe list of predicted signs. Video encoder 22 may iterate this processcontinues for each interval in descending order until the number ofpredicted signs reaches the pre-defined maximum number of predictedsigns, or video encoder 22 has checked all available intervals. If videoencoder 22 determines that a coefficient has a level less than t_(N),video encoder 22 may not consider the sign to be a predicted sign. (600,+∞), [300, 600) represents a common or even typical setting. Videoencoder 22 may not consider the signs of dequantized coefficient levelsbeing less than 300 as predicted signs, in this example.

Some aspects of this disclosure are directed to determining the CABACcontext for coding coefficient signs. That is, according to thesetechniques, video encoder 22 may define context values for the signs, bysetting different contexts for different intervals of coefficientlevels. Again, intervals can vary based on various features. The contextvalue that video encoder 22 uses encoding coefficient signs may dependon the dequantized level (e.g., the absolute value) of the correspondingcoefficient. Video encoder 22 may pre-define several intervals, and maydetermine one of the context values using the index of interval to whichthe dequantized level value belongs.

An example setting of intervals are (0, 600), [600, +∞), or (0, 600],(600, +∞). With this interval setting, if the coefficient level falls inthe range of (0, 600), then video encoder 22 may set the context valueto zero (0). Otherwise (i.e., if the coefficient level falls outside ofthe (0, 600) range), video encoder 22 may set the context value to one(1). In various examples, video encoder 22 (or one or more componentsthereof) may implement the CABAC context determination aspects of thisdisclosure to determine the intervals as being dependent onpreviously-encoded information, including, but not limited to, one ormore of the QP for the block, the block area (“size”), block width, orblock height.

Various aspects of this disclosure are directed to a searching order fora hypothesis, such as for one of the two hypotheses that video encoder22 may form during a hypothesis reconstruction for a block. Again, eachhypothesis consists of one possible combination of candidate signvalues. For example, if video encoder 22 predicts two signs, videoencoder 22 may need to form a total of four hypotheses. That is, in thetwo-sign scenario, video encoder 22 may form four possible combinationsof candidate sign values, including the following:

Hypothesis index 0: {0, 0}

Hypothesis index 1: {0, 1}

Hypothesis index 2: {1, 0}

Hypothesis index 3: {1, 1},

where the first digit indicates the first predicted sign, and the seconddigit indicates the second predicted sign. In the examples above, zero(0) indicates a positive sign, while one (1) indicates a negative sign.The combination of candidate sign values in each hypothesis correspondsto the binarized representation of the hypothesis index. For example, a{1, 1} combination yields the hypothesis index of three, because the 2̂0bit and the 2̂1 bit are both set to ‘1’ values. If a coding devicesearches the hypotheses in an increasing order or a decreasing order,there can be more than one change of sign values between two neighborhypotheses. As shown in the example above, the transition fromhypothesis index 1 to hypothesis index 2, both signs change in value(e.g., by toggling). As such boundary sample changes that involvetoggling of both signs may need to be accounted for in hypothesisreconstruction.

To take into account the two-sign toggling of certain boundary samples,video encoder 22 may implement the hypothesis searching order aspects ofthis disclosure according to one or more of the techniques describedbelow. To reduce the complexity, according to aspects of thisdisclosure, video encoder 22 may implement the searching order followinga Gray code. According to the Gray code-based sign assignment, videoencoder 22 may search the four hypotheses according to the followingsign-index assignments:

Hypothesis index 0: {0, 0}

Hypothesis index 1: {0, 1}

Hypothesis index 2: {1, 1}

Hypothesis index 3: {1, 0}

As shown above, the Gray code-based hypothesis assignment of thisdisclosure results in only one sign value being toggled whentransitioning between two neighboring hypotheses. In this way, based onthe boundary sample values generated for a previous hypothesis, videoencoder 22 may, according to aspects of this disclosure, generate theboundary sample values for current hypothesis by always consider thechange (toggle) incurred by only one sign. In this way, video encoder 22may implement the techniques of this disclosure to reduce the complexityof hypothesis generation, and to regularize the cost calculationprocess, by incorporating the Gray code into the hypothesis searchingorder.

FIG. 5 is a block diagram illustrating an example video decoder 30 thatis configured to implement the techniques of this disclosure. FIG. 5 isprovided for purposes of explanation and is not limiting on thetechniques as broadly exemplified and described in this disclosure. Forpurposes of explanation, this disclosure describes video decoder 30 inthe context of HEVC coding. However, the techniques of this disclosuremay be applicable to other coding standards or methods.

In the example of FIG. 5, video decoder 30 includes an entropy decodingunit 150, video data memory 151, a prediction processing unit 152, aninverse quantization unit 154, an inverse transform processing unit 156,a reconstruction unit 158, a filter unit 160, and a decoded picturebuffer 162. Prediction processing unit 152 includes a motioncompensation unit 164 and an intra-prediction processing unit 166. Inother examples, video decoder 30 may include more, fewer, or differentfunctional components.

Video data memory 151 may store encoded video data, such as an encodedvideo bitstream, to be decoded by the components of video decoder 30.The video data stored in video data memory 151 may be obtained, forexample, from computer-readable medium 16, e.g., from a local videosource, such as a camera, via wired or wireless network communication ofvideo data, or by accessing physical data storage media. Video datamemory 151 may form a coded picture buffer (CPB) that stores encodedvideo data from an encoded video bitstream. Decoded picture buffer 162may be a reference picture memory that stores reference video data foruse in decoding video data by video decoder 30, e.g., in intra- orinter-coding modes, or for output. Video data memory 151 and decodedpicture buffer 162 may be formed by any of a variety of memory devices,such as dynamic random access memory (DRAM), including synchronous DRAM(SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or othertypes of memory devices. Video data memory 151 and decoded picturebuffer 162 may be provided by the same memory device or separate memorydevices. In various examples, video data memory 151 may be on-chip withother components of video decoder 30, or off-chip relative to thosecomponents. Video data memory 151 may be the same as or part of storagemedia 28 of FIG. 1.

Video data memory 151 receives and stores encoded video data (e.g., NALunits) of a bitstream. Entropy decoding unit 150 may receive encodedvideo data (e.g., NAL units) from video data memory 151 and may parsethe NAL units to obtain syntax elements. Entropy decoding unit 150 mayentropy decode entropy-encoded syntax elements in the NAL units.Prediction processing unit 152, inverse quantization unit 154, inversetransform processing unit 156, reconstruction unit 158, and filter unit160 may generate decoded video data based on the syntax elementsextracted from the bitstream. Entropy decoding unit 150 may perform aprocess generally reciprocal to that of entropy encoding unit 118. Invarious examples, entropy decoding unit 150 may be configured toimplement one or more techniques of this disclosure.

In accordance with some examples of this disclosure, entropy decodingunit 150 may determine a tree structure as part of obtaining the syntaxelements from the bitstream. The tree structure may specify how aninitial video block, such as a CTB, is partitioned into smaller videoblocks, such as coding units. In accordance with one or more techniquesof this disclosure, for each respective non-leaf node of the treestructure at each depth level of the tree structure, there are aplurality of allowed splitting patterns for the respective non-leaf nodeand the video block corresponding to the respective non-leaf node ispartitioned into video blocks corresponding to the child nodes of therespective non-leaf node according to one of the plurality of allowablesplitting patterns.

In addition to obtaining syntax elements from the bitstream, videodecoder 30 may perform a reconstruction operation on a non-partitionedCU. To perform the reconstruction operation on a CU, video decoder 30may perform a reconstruction operation on each TU of the CU. Byperforming the reconstruction operation for each TU of the CU, videodecoder 30 may reconstruct residual blocks of the CU.

As part of performing a reconstruction operation on a TU of a CU,inverse quantization unit 154 may inverse quantize, i.e., de-quantize,coefficient blocks associated with the TU. After inverse quantizationunit 154 inverse quantizes a coefficient block, inverse transformprocessing unit 156 may apply one or more inverse transforms to thecoefficient block in order to generate a residual block associated withthe TU. For example, inverse transform processing unit 156 may apply aninverse DCT, an inverse integer transform, an inverse Karhunen-Loevetransform (KLT), an inverse rotational transform, an inverse directionaltransform, or another inverse transform to the coefficient block.

If a PU is encoded using intra prediction, intra-prediction processingunit 166 may perform intra prediction to generate predictive blocks ofthe PU. Intra-prediction processing unit 166 may use an intra predictionmode to generate the predictive blocks of the PU based on samplesspatially-neighboring blocks. Intra-prediction processing unit 166 maydetermine the intra prediction mode for the PU based on one or moresyntax elements obtained from the bitstream.

If a PU is encoded using inter prediction, entropy decoding unit 150 maydetermine motion information for the PU. Motion compensation unit 164may determine, based on the motion information of the PU, one or morereference blocks. Motion compensation unit 164 may generate, based onthe one or more reference blocks, predictive blocks (e.g., predictiveluma, Cb and Cr blocks) for the PU.

Reconstruction unit 158 may use transform blocks (e.g., luma, Cb and Crtransform blocks) for TUs of a CU and the predictive blocks (e.g., luma,Cb and Cr blocks) of the PUs of the CU, i.e., either intra-predictiondata or inter-prediction data, as applicable, to reconstruct the codingblocks (e.g., luma, Cb and Cr coding blocks) for the CU. For example,reconstruction unit 158 may add samples of the transform blocks (e.g.,luma, Cb and Cr transform blocks) to corresponding samples of thepredictive blocks (e.g., luma, Cb and Cr predictive blocks) toreconstruct the coding blocks (e.g., luma, Cb and Cr coding blocks) ofthe CU.

Filter unit 160 may perform a deblocking operation to reduce blockingartifacts associated with the coding blocks of the CU. Video decoder 30may store the coding blocks of the CU in decoded picture buffer 162.Decoded picture buffer 162 may provide reference pictures for subsequentmotion compensation, intra prediction, and presentation on a displaydevice, such as display device 32 of FIG. 1. For instance, video decoder30 may perform, based on the blocks in decoded picture buffer 162, intraprediction or inter prediction operations for PUs of other CUs.

According to some aspects of this disclosure, video decoder 30 may use apre-stored set of reconstructed residual samples to perform hypothesisreconstruction. Techniques of this disclosure by which video decoder 30uses pre-stored reconstructed residual samples are generally referred toherein as “template-based reconstruction” techniques, and may representdecoder-side operations corresponding to the template-basedreconstruction techniques described above with respect to video encoder22. In some instances of template-based reconstruction according toaspects of this disclosure, video decoder 30 may store one or morefully-reconstructed and/or partially-reconstructed residual blocks, andthen use some combination (e.g., a linear combination) of the storedinformation to form a template.

For instance, the pre-stored reconstructed residual samples may bestored to any memory accessible (e.g., via direct or indirectcommunicative coupling) to processing circuitry of video decoder 30,such as storage media 28 of destination device 14 and/or video datamemory 151. In turn, video decoder 30 may use the template to derive allof the possible reconstructed values. Each stored set of residual blocksmay be pre-calculated using a pre-defined transform coefficient block.To perform hypothesis reconstruction, video decoder 30 may identify aselected group of fully- and/or partially-reconstructed residual blocksthe stored set, and generate/estimate the hypothesis reconstructionusing some combination the reconstructed residual blocks identified inthis fashion.

As discussed above, video decoder 30 (and/or components thereof) may beconfigured, according to aspects of this disclosure, to mitigate variousissues associated with existing sign prediction technology. According tosome examples of the template-based reconstruction techniques of thisdisclosure, video decoder 30 may form each reconstructed block of thestored set using a pre-defined quantized transform coefficient block(TB), a given dequantization parameter, and a given primary transformand/or a given secondary transform. An example is discussed below withrespect to a particular coefficient, for which the position is denotedas f(x,y) in an M×N TB, where 0<=x<M, 0<=y<N. Given the inversetransform method (including primary transform and/or secondarytransform) and dequantization parameter, video decoder 30 may generate aquantized coefficient block (denoted as B) by setting f(x,y) to acertain value, and by setting all other coefficients to a differentvalue from the value assigned to f(x,y). For example, video decoder 30may set the value of f(x,y) to one (1), and may set all othercoefficients to zero (0).

In turn, video decoder 30 may pre-calculate a fully- orpartially-reconstructed residual block (denoted as R) by applying thegiven dequantization and inverse transform on the quantized coefficientblock B. Video decoder 30 may pre-store the reconstructed residual blockR to a memory (e.g., to storage media 28 of destination device 14 and/orto video data memory 101). By pre-storing the partially- orfully-reconstructed block R, video decoder 30 may leverage existingresources (e.g., in the form of storage media 20 and/or video datamemory 151) to cache pre-calculated information that can be used toimprove the efficiency of calculations to be performed at a later stage.In this manner, during sign prediction operations, if the sign ofcoefficient f(x,y) needs to be predicted in an M×N TB with the sameinverse transform and dequantization parameter, the techniques of thisdisclosure enable video decoder 30 to avoid performing an inversetransform for f(x,y). Instead, video decoder 30 may scale the pre-storedreconstructed residual block R by a factor of |f(x,y)| and then use thescaled version of the pre-stored reconstructed residual block R toestimate the reconstructed block.

In another example of the template-based reconstruction techniques ofthis disclosure, video decoder 30 may pre-store a dequantized version ofthe TB. That is, in these examples, video decoder 30 may reconstruct thecurrent block using dequantized versions of the (fully or partially)reconstructed blocks stored to storage media 28. In other words,according to these examples, video decoder 30 may reconstruct thecurrent block using a pre-defined dequantized TB, and one or both of agiven primary transform or a given secondary transform.

One use-case scenario of the dequantized TB-based implementations of thetemplate-based reconstruction techniques of this disclosure is describedbelow with respect to a particular coefficient f(x,y) in an M×N TB,where 0<=x<M, 0<=y<N, given the inverse transform method (includingprimary transform and/or secondary transform). In this example, videodecoder 30 may generate a dequantized coefficient block B by settingf(x,y) to a certain value, and by setting all other coefficients toanother value. For instance, video decoder 30 may set f(x,y) to a valueof one (1), and may set all of the remaining coefficients to a value ofzero (0). In turn, video decoder 30 may pre-calculate a fully orpartially reconstructed residual block R by applying the given inversetransform on B, and may pre-store the reconstructed residual block R toa memory (e.g., to storage media 28 and/or video data memory 151).

In this way, during sign prediction, if the sign of the quantizedcoefficient f(x,y) needs to be predicted in an M×N TB with the sameinverse transform, the techniques of this disclosure enable videodecoder 30 and/or video decoder 30 to avoid doing inverse transform forf(x,y). Instead, video decoder 30 may first dequantize the quantizedcoefficient f(x,y) to be f(x,y)′ (to denote the dequantizedcoefficient), then video decoder 30 ma scale the pre-storedreconstructed residual block R by a factor of |f(x,y)′|. Video decoder30 may then use the scaled version (i.e., scaled using the dequantizedvalue) of the pre-stored dequantized reconstructed residual block R toestimate the reconstructed block. As illustrated in the discussionabove, video decoder 30 may implement the template-based reconstructiontechniques of this disclosure by optionally using dequantized TBinformation for the pre-stored template, with varying computationcomplexity arising from the optional use or non-use of thedequantization operations.

In some implementations of the template-based reconstruction techniquesof this disclosure, video decoder 30 may pre-store the set of partially-and/or fully-reconstructed residual blocks with fractional precision. Insome examples, video decoder 30 may first scale the set of reconstructedresidual blocks with a pre-defined factor (e.g., 2, 4, 8, 16, etc.), andthen round the result to the nearest integer, and store the rounded,scaled result to storage media 28 and/or video data memory 151. In thisway, video decoder 30 may enhance the precision of the pre-storedreconstructed residual blocks. Moreover, in both the fractionalprecision implementations and the scaling/rounding based implementationsdescribed above, video decoder 30 may avail of the benefits of thetemplate-based reconstruction techniques of this disclosure, whilereducing the memory requirements with respect to storage media 28 and/orvideo data memory 151.

According to some implementations of the template-based reconstructiontechniques of this disclosure, video decoder 30 may pre-store the set offully- and/or partially-reconstructed residual blocks with clippedvalues. In some instances, video decoder 30 may clip the values of thereconstructed residual blocks before storage, such that the clippedvalues fit a restricted bit range of representation. For example, byclipping the values using constraints of a minimum value (Vmin) ofnegative eight (−8) and a maximum value (Vmax) of seven (7), videodecoder 30 may form the entries of the pre-stored reconstructed residualvalues such that all of the pre-stored reconstructed residual values canbe represented using, at most, four (4) bits. In various use-casescenarios, video decoder 30 may utilize one or more of the factionalprecision implementation, the scaling/rounding implementation, or theclipping-based implementations of the template-based reconstructiontechniques individually or in any combination, in accordance withvarious aspects of this disclosure.

In some examples of the template-based reconstruction techniques of thisdisclosure, video decoder 30 may use a linear combination of certainidentified reconstructed residual blocks of the stored reconstructedresidual blocks to reconstruct the current TB. For example, the linearcombination may represent a weighted sum of the identified storedreconstructed residual blocks. To calculate the weighted sum, videodecoder 30 may determine the respective weights using the magnitude(also referred to as ‘level value’) of a quantized or dequantized (asthe case may be) transform coefficient for which the sign is beingpredicted. By using a weighted sum of multiple stored reconstructedresidual blocks, video decoder 30 may improve hypothesis precision byutilizing data from multiple previously-reconstructed blocks.

Video decoder 30 may, in some examples, implement template-basedreconstruction techniques of this disclosure in a manner that makes usof offline computation. Because different transform and/or quantizationmethods may be available to video decoder 30, video decoder 30 may needto store multiples of reconstructed blocks for a single quantizedcoefficient block. To mitigate storage requirements and to ease theresource consumption of accessing the pre-calculated residual blocks,video decoder 30 may avail of offline computation capabilities. Forinstance, video decoder 30 may perform a merging process on thepre-stored residual blocks, such that multiple residual blocks aremerged and represented by one residual block. In some examples, videodecoder 30 may pre-define a function ƒ to derive one pre-stored residualblock from another pre-stored residual block, such that only one ofseveral residual blocks may need to be stored. That is, in theseexamples, video decoder 30 may store one residual block, and may deriveother residual blocks using the pre-defined function ƒ and the singlestored residual block.

In some implementations of the template-based reconstruction techniquesof this disclosure, video decoder 30 may store pre-calculated residualblocks to off-chip memory (also referred to as “external” memory) ofsource device 12. The total size of the set of pre-calculated residualblocks may, in some instances, be large, and therefore may be difficultto store in on-chip memory. To continue using the template-basedreconstruction techniques of this disclosure while operating within theavailable storage constraints, video decoder 30 may pre-store the set ofpre-calculated residual blocks to off-chip memory, which, in someexamples, tends to represent a cheaper memory than on-chip memory. Inthese implementations, video decoder 30 may only load the residualblocks, as needed, to on-chip memory. For instance, video decoder 30 mayload the residual blocks once per bigger block or bigger picture area,such as for a largest coding unit (LCU).

According to some implementations of the template-based reconstructiontechniques of this disclosure, video decoder 30 may use each of thepre-stored set of partial residual blocks as a template for signprediction. In some examples, video decoder 30 may store only theleftmost and topmost residual pixels are stored for the pre-calculatedresidual blocks. In one example, video decoder 30 may store the top-leftresidual pixel, in addition to one or more residual pixels of the toprow (starting from the residual pixel positioned immediately right ofthe top-left residual pixel) and/or one or more residual pixels of theleftmost column (starting from the residual pixel positioned immediatelybelow of the top-left residual pixel). In some examples, video decoder30 may apply the template-based sign prediction process described inHenry in conjunction with the pre-stored set of partial residual blocks(of this disclosure) in such a way that the template is not generated bydoing partial inverse transform on the fly. Instead, video decoder 30may generate the template by accessing the pre-stored set of partialresidual blocks, and performing a scaling operation.

Video decoder 30 may, in some examples, be configured according toaspects of this disclosure to perform selective sign prediction. Forinstance, video decoder 30 may perform selective sign prediction withrespect to lower-frequency transform coefficients. Video decoder 30 mayalso perform selective sign prediction to avoid sign prediction foroutlier block sizes (e.g. very large or very small block sizes).Examples of outlier dimensions that video decoder 30 may use inaccordance with the techniques of this disclosure are discussed below.To perform the selective sign prediction techniques of this disclosure,video decoder 30 may determine various conditions under which signprediction is to be performed or alternatively skipped. Variousconditions for the selective sign selection aspects of this disclosureare described in the following paragraphs.

Video decoder 30 may apply several of the conditions for the selectivesign prediction aspects of this disclosure on a block-wide basis. Theseconditions may relate to various features including, but not limited to,the block width, block height, block area (also referred to as “size”),transform coefficient frequency locations, last position, intraprediction mode, whether only a primary transform is used to code ablock, whether only particular transform functions or certain transformsubset are used for coding the block, inverse transform method, whetherthe block is intra coded or inter coded blocks, or any other decodedinformation of the current and/or neighbor block(s).

According to some implementations of the selective sign predictiontechniques of this disclosure, video decoder 30 may not apply signprediction if either the width of the currently-decoded block or theheight of the currently-decoded block is larger than a pre-definedthreshold. In contrast, according to existing sign predictiontechnology, sign prediction would be applied regardless of the blockwidth or height, as the case may be. In some examples, the threshold maybe configurable, and video decoder 30 may receive the threshold in anencoded video bitstream signaled by video encoder 22. In variousexamples, video decoder 30 may set the threshold (using the signaledinformation or otherwise) to values including, but not limited to 4, 8,16, 32, or 64.

In some examples of the selective sign prediction techniques of thisdisclosure, video decoder 30 may configure the threshold value to bedependent on coding modes according to which the block was encoded (andreciprocally, that video decoder 30 may use to decode the encodedblock). For example, video decoder 30 may configure the threshold to bedifferent for intra-coded blocks and inter-coded blocks. In one example,video decoder 30 may set the threshold value to 32 for intra-codedblocks, while setting the threshold value to 64 for inter-coded block.In some examples, video decoder 30 may not apply sign prediction if theminimum or maximum of block width or block height exceeds a threshold.

According to some examples of the sign prediction techniques of thisdisclosure, video decoder 30 may not apply sign prediction for transformcoefficients that are not located in the top-left K×L part of a TB. K×Lmay represent a square sub-block of the TB, such as a sub-block having adimensionality of 2×2, 3×3, 4×4 or 8×8, etc. In contrast, according toexisting sign prediction technology, sign prediction would be appliedfor transform coefficients positioned outside of the top-left K×Lportion of the TB. In some examples, the values of K and/or L mayfurther depend on coding modes and/or quantization parameters. In thespecific example of the K×L portion of the TB representing a 2×2sub-block, there may be only four (4) possible sign predictions.

In some implementations of the selective sign prediction techniques ofthis disclosure, video decoder 30 may limit the inverse transformperformed in the sign prediction process to certain inversetransform(s). For example, video decoder 30 may impose the inversetransform constraint while generating a hypothesis during the hypothesisprediction process. The selected transform(s) can be different from theinverse transform which video decoder 30 applies to generate the actualreconstructed block after deriving the actual coefficient signs.

In one such example, video decoder 30 may only use discrete cosinetransform type-II (DCT-II) as the primary inverse transform during signprediction. In another such example, during sign prediction, videodecoder 30 may replace the secondary transform with a transform using apre-defined permutation matrix. The pre-defined permutation matrix mayinclude entities of “+1” and “−1”. In another example, video decoder 30may not use a secondary transform during sign prediction.

According to some examples of the selective sign prediction techniquesof this disclosure, video decoder 30 may apply sign prediction only forthose encoded blocks that were encoded using only a primary transform.By limiting sign prediction in this manner, video decoder 30 may reducecoding complexity, such that the complexity of the inverse transform maybe smaller than the complexity arising from having a primary and asecondary transform applied together. In another example, video decoder30 may apply sign prediction only for those encoded blocks that wereencoded using a certain transform core or certain transform set/subset.Limiting sign prediction in this manner may yield potential benefitsbecause it is possible that sign prediction may not work efficiently forall transform basis/cores, and coding complexity can thereby be reducedby limiting the application of sign prediction in this way.

Some aspects of this disclosure are directed to techniques for costfunction measurement. According to the cost function measurement aspectsof this disclosure, video decoder 30 may utilize previously-decodedinformation (obtained via the decoding loop) to measure the costfunction for each hypothesis reconstruction. The previously-decodedinformation that video decoder 30 may leverage for cost functionmeasurement includes, but is not limited to, the width of the currentblock, the height of the current block, the current block's area (alsoreferred to as “size” herein), transform coefficient frequencylocations, a last position, an intra prediction mode used for encodingthe block, whether the evaluated block(s) are intra-coded orinter-coded, or any decoded information of the current and/or neighborblock(s).

In some examples, video decoder 30 (or one or more components thereof)may implement the cost function measurement aspects of this disclosureby measuring a difference between reconstructed samples of theneighboring block(s) and the reconstructed samples of the current blocklocated along the boundary. The measured difference is also referred toherein as a “gradient” to illustrate that a difference can be expressedas the slope at a given point of a curve. That is, instead of measuringthe first order information representing the sample values, videodecoder 30 may implement the cost function measurement techniques ofthis disclosure to measure second order (difference or gradient)information that is comparative between the neighboring reconstructedsamples and the reconstructed samples of the current block positionedalong the boundary.

According to this implementation, video decoder 30 may generateestimated boundary residual samples by subtracting the neighboringreconstructed samples from the prediction samples located at the currentblock boundary (or vice versa). That is, video decoder 30 may use thesubtraction operation(s) to obtain the second order informationrepresented by the difference between the estimated boundary residualsamples and the candidate boundary residual samples. For each candidatecombination of signs, the generated residual samples represent a set of“candidate residual samples.” By using the second order gradientinformation between the neighboring residual samples and theboundary-interfacing residual samples of the current block, videodecoder 30 may avoid instances of duplicate reconstruction processes, inwhich the same prediction samples are used to produce candidate residualsamples multiple times. In this way, techniques of this disclosureenable video decoder 30 to reduce computing resource consumption thatwould otherwise be expended for redundant candidate residual samplegeneration. In this way, video decoder 30 may also reduce bandwidthrequirements by reducing the need for video encoder 22 to signal theredundant candidate residual sample information in an encoded videobitstream.

In some examples of the cost function measurement techniques of thisdisclosure, video decoder 30 may be configured to measure the pixelvalue difference (also referred to herein as a “pixel gradient value”)along an intra-prediction direction to measure the cost function. In onesuch example, video decoder 30 may pre-select a partial subset ofallowed intra prediction directions, and may map each allowed intraprediction direction to one of the pre-selected intra directions whichcan be used to approximate the intra prediction direction. In thisexample, video decoder 30 may measure the pixel value difference alongthe mapped direction in the cost function calculation.

In cases where the intra prediction direction is not available to videodecoder 30 (e.g., if one of the evaluated blocks is inter-coded), videodecoder 30 may instead derive an intra prediction mode. Video decoder 30may derive the intra prediction mode from the spatial neighboring codedblocks, or a block identified by a motion vector in a coded picture. Ifthe intra prediction direction points to a fractional position, videodecoder 30 may apply an interpolation process to derive the value of thefractional position. Video decoder 30 may apply the interpolation filterused for generating the intra-predicted block, or may use a different(e.g., simpler) interpolation filter instead.

According to some of the cost function measurement aspects of thisdisclosure, video decoder 30 may measure the cost function using asecond or higher order (N-th order, where ‘N’ represents an integervalue of two or greater) order of the pixel value difference. Forinstance, video decoder 30 may use multiple reference lines (e.g., twoor more) to measure the cost value. The number of reference lines thatvideo decoder 30 uses for measuring the cost value may depend on codedinformation, including, but not limited to, the intra prediction modeand/or neighboring reconstructed values. Video decoder 30 may, in someinstances, set the maximum number of reference lines to be equal to therequired number of reference lines for intra prediction.

According to some implementations of the cost function measurementaspects of this disclosure, video decoder 30 may exclude some pixelsfrom the cost function measurement operations, depending on whether oneor more given criteria are satisfied. In one example, video decoder 30uses a criterion of whether the absolute value of a first order, secondorder (gradient), or N-th order derivative is less than a giventhreshold value. In another example, video decoder 30 uses a criterionof whether the neighboring reconstructed samples used to calculate theN-th order derivative is available. In another example still, videodecoder 30 uses a criterion of whether identity transform or transformskip is applied to the horizontal or vertical transform.

According to some implementations of the cost function measurementaspects of this disclosure, video decoder 30 may, when measuring thecost using neighboring reconstructed values, calculate the cost valuesfor multiple directions. That is, video decoder 30 may calculate thecost values for multiple directions instead of calculating the firstorder, second order (gradient), or N-th order differences along a fixedor pre-defined direction (e.g., horizontal direction for a left blockboundary and vertical direction for a top block boundary). Rather,according to these implementations, video decoder 30 calculates the costvalues for multiple directions, and uses the minimum cost for measuringthe cost.

In some examples of the cost function measurement techniques of thisdisclosure, for relatively larger block sizes, video decoder 30 maycalculate the cost function using a downsampled hypothesisreconstruction. By using a downsampled hypothesis reconstruction forcost function measurement, video decoder 30 may reduce the complexity ofcost function application with respect to larger block sizes. Accordingto some of the cost function measurement techniques of this disclosure,video decoder 30 may measure the pixel value difference only for partialpixels in the reconstructed block and neighbor block(s). Partial pixelsmay be, for example, corner pixels. Errors typically tend to be largerfor corner pixels. In another example, video decoder 30 may exclude thecorner pixels from the difference measure based on the corner pixelsbeing potential outliers, or may assign different weights to theselected pixels.

According to some implementations of the cost function measurementaspects of this disclosure, video decoder 30 may determine the number ofpredicted signs (e.g., 0, 1, 2, 3, 4, 5, 6, etc.) in a way that thenumber of predicted signs depends on previously-decoded information.Non-limiting examples of previously-decoded information that videodecoder 30 may use in determining the number of predicted signs include,but are not limited to, intra prediction mode, block width/height, blocksize (area), QP, color components, temporal layer, whether the block isintra-coded or inter coded, whether transform skip or identity transformis applied, whether non-separable secondary transform is applied,quantized coefficients, the presence/value of an enhanced multipletransform (EMT) index, or the presence/value of a position-dependentprediction combination (PDPC) flag or index.

According to some implementations of the cost function measurementaspects of this disclosure, video decoder 30 may, when multiple ‘M’signs are to be predicted, perform sign prediction in a unit or groupmanner. That is, video decoder 30 may perform sign prediction for agroup, instead of testing all the possible 2̂M hypothesisreconstructions, where the ‘M’ signs are divided into ‘N’ groups. Videodecoder 30 may predict each group of signs by setting the remaininggroups of signs as previously-predicted values or default value, if theremaining group of signs have not yet been predicted. Video decoder 30may iterate the group-based sign prediction process for multiple roundsuntil a pre-defined limitation is met. Video decoder 30 may predict agroup of signs with larger transform coefficient levels prior topredicting a group of signs with relatively lower coefficient levels.Video decoder 30 may terminate the iterations when the predicted signsfor each group no longer change, or the iteration number has met apre-defined threshold value, or the cost difference between iterations(for example, succeeding iterations) has stopped exceeding a certainthreshold.

The following paragraphs describe aspects of this disclosure thatgenerally relate to the selection of coefficient signs to be predicted.According to the coefficient sign selection aspects of this disclosure,video decoder 30 may select predicted transform coefficients byutilizing already-available decoded information (e.g., via a decodingloop implemented by video decoder 30). The previously-decodedinformation that video decoder 30 may use for coefficient sign selectionincludes, but is not limited to, one or more of the width of the currentblock, the height of the current block, the area (also referred to as“size”) of the current block, transform coefficient frequency locationswithin the current block, last position, an intra prediction modeaccording to which the current block the current and/or neighboringblock(s) were encoded, whether sign data hiding is applied with respectto the current and/or neighboring block(s), whether the current and/orneighboring block(s) were intra-coded or inter-coded, or any decodedinformation of the current and/or neighbor block(s). In variousexamples, video decoder 30 (or one or more components thereof) mayimplement the coefficient sign selection aspects of this disclosure suchthat the maximum number of predicted signs depends on whether thecurrent block is inter-coded or intra-coded. In some examples, videodecoder 30 may set the maximum number of predicted signs based oninverse transform method (e.g., which primary and/or secondarytransform) is applied for dequantizing the block.

In some examples, video decoder 30 may apply sign hiding even when signprediction is applied the block (TB). In these examples, video decoder30 may hide the sign of the last nonzero coefficient in the forwardscanning order, instead of hiding the first nonzero coefficient in theforward scanning order. Furthermore, in some such examples, videodecoder 30 may not perform sign data hiding at the CG-level. That is, inthese instances, consecutive nonzero coefficients, even if located indifferent CGs, may be jointly considered for sign data hiding.

In some examples, video decoder 30 may pre-define several thresholdvalues (denoted by t₀, t₁, . . . , t_(N)) where t₀>t₁> . . . >t_(N). Inthese examples, video decoder 30 may derive several intervals as (t₀,+∞), [t₁, t₀), [t_(N), t_(N-1)) in descending order. Moreover, videodecoder 30 may compare dequantized coefficient levels to each one of thegiven intervals until video decoder 30 identifies or detects oneinterval which covers the dequantization coefficient level. Videodecoder 30 may identify the signs of coefficients as predicted signs,according to the descending order of the threshold values.

For instance, video decoder 30 may first place the signs forcoefficients in the interval (t₀, +∞) into the list of predicted signs,and then place the signs for coefficients in the interval [t₁, t₀) intothe list of predicted signs. Video decoder 30 may iterate this processcontinues for each interval in descending order until the number ofpredicted signs reaches the pre-defined maximum number of predictedsigns, or video decoder 30 has checked all available intervals. If videodecoder 30 determines that a coefficient has a level less than t_(N),video decoder 30 may not consider the sign to be a predicted sign. (600,+∞), [300, 600) represents a common or even typical setting. Videodecoder 30 may not consider the signs of dequantized coefficient levelsbeing less than 300 as predicted signs, in this example.

Some aspects of this disclosure are directed to determining the CABACcontext for coding coefficient signs. That is, according to thesetechniques, video decoder 30 may define context values for the signs, bysetting different contexts for different intervals of coefficientlevels. Again, intervals can vary based on various features. The contextvalue that video decoder 30 uses encoding coefficient signs may dependon the dequantized level (e.g., the absolute value) of the correspondingcoefficient. Video decoder 30 may pre-define several intervals, and maydetermine one of the context values using the index of interval to whichthe dequantized level value belongs.

A common or even typical setting of intervals are (0, 600), [600, +∞),or (0, 600], (600, +∞). With this interval setting, if the coefficientlevel falls in the range of (0, 600), then video decoder 30 may set thecontext value to zero (0). Otherwise (i.e., if the coefficient levelfalls outside of the (0, 600) range), video decoder 30 may set thecontext value to one (1). In various examples, video decoder 30 (or oneor more components thereof) may implement the CABAC contextdetermination aspects of this disclosure to determine the intervals asbeing dependent on previously-decoded information, including, but notlimited to, one or more of the QP for the block, the block area(“size”), block width, or block height.

Various aspects of this disclosure are directed to a searching order fora hypothesis, such as for one of the two hypotheses that video decoder30 may form during a hypothesis reconstruction for a block. Again, eachhypothesis consists of one possible combination of candidate signvalues. For example, if video decoder 30 predicts two signs, videodecoder 30 may need to form a total of four hypotheses. That is, in thetwo-sign scenario, video decoder 30 may form four possible combinationsof candidate sign values, including the following:

Hypothesis index 0: {0, 0}

Hypothesis index 1: {0, 1}

Hypothesis index 2: {1, 0}

Hypothesis index 3: {1, 1},

where the first digit indicates the first predicted sign, and the seconddigit indicates the second predicted sign. In the examples above, zero(0) indicates a positive sign, while one (1) indicates a negative sign.The combination of candidate sign values in each hypothesis correspondsto the binarized representation of the hypothesis index. For example, a{1, 1} combination yields the hypothesis index of three, because the 2̂0bit and the 2̂1 bit are both set to ‘1’ values. If a coding devicesearches the hypotheses in an increasing order or a decreasing order,there can be more than one change of sign values between two neighborhypotheses. As shown in the example above, the transition fromhypothesis index 1 to hypothesis index 2, both signs change in value(e.g., by toggling). As such boundary sample changes that involvetoggling of both signs may need to be accounted for in hypothesisreconstruction.

To take into account the two-sign toggling of certain boundary samples,video decoder 30 may implement the hypothesis searching order aspects ofthis disclosure according to one or more of the techniques describedbelow. To reduce the complexity, according to aspects of thisdisclosure, video decoder 30 may implement the searching order followinga Gray code. According to the Gray code-based sign assignment, videodecoder 30 may search the four hypotheses according to the followingsign-index assignments:

Hypothesis index 0: {0, 0}

Hypothesis index 1: {0, 1}

Hypothesis index 2: {1, 1}

Hypothesis index 3: {1, 0}

As shown above, the Gray code-based hypothesis assignment of thisdisclosure results in only one sign value being toggled whentransitioning between two neighboring hypotheses. In this way, based onthe boundary sample values generated for a previous hypothesis, videodecoder 30 may, according to aspects of this disclosure, generate theboundary sample values for current hypothesis by always consider thechange (toggle) incurred by only one sign. In this way, video decoder 30may implement the techniques of this disclosure to reduce the complexityof hypothesis generation, and to regularize the cost calculationprocess, by incorporating the Gray code into the hypothesis searchingorder.

FIG. 6 is a flowchart illustrating an example process 180 by which avideo coding device may perform template-based reconstruction techniquesof this disclosure as part of a hypothesis reconstruction for a currentblock. Although a variety of video coding devices may perform process180, FIG. 6 is described as being performed by video decoder 30 (andmore specifically, processing circuitry thereof), as a non-limitingexample. Process 180 may begin when processing circuitry of videodecoder 30 identifies a set of pre-stored residual samples (182). Forinstance, the processing circuitry of video decoder 30 may identify theset of pre-stored residual samples within pre-stored video dataavailable from video data memory 151. The pre-stored residual samplesmay, in some examples, be included in a full pre-stored block, or inpartial blocks stored to video data memory 151.

In some examples, the pre-stored residual samples may representhard-coded constants, may not belong to previously-reconstructedresidual blocks. Rather, in these examples, the pre-stored residualsamples may represent an approximation of reconstructed residual data,and not actual reconstructed residual data. In turn, the processingcircuitry of video decoder 30 may combine the identified set ofpre-stored residual samples (184). The processing circuitry of videodecoder 30 may form the combination of the identified set ofpreviously-reconstructed residual blocks by implementing a variety ofcombinatorial logic. In some examples, the processing circuitry of videodecoder 30 may form a linear combination of the identifiedpreviously-reconstructed residual blocks. For instance, to form a linearcombination of the identified set of previously-reconstructed residualblocks, the processing circuitry of video decoder 30 may calculate aweighted sum of the transform coefficient levels of the block.

Additionally, the processing circuitry of video decoder 30 may obtainmultiple hypothesis reconstructions for the current block, using thecombination of pre-stored residual samples (186). For instance, theprocessing circuitry of video decoder 30 may apply different signs ordifferent combinations of signs to the combination of pre-storedresidual samples to form each respective hypothesis reconstruction. Inthis way, the processing circuitry of video decoder 30 may implement thetechniques of this disclosure to use pre-stored residual samples (whichare also referred to as “stored templates”) to perform hypothesisreconstruction with respect to a current block.

In turn, the processing circuitry of video decoder 30 may deriverespective cost functions for multiple hypothesis reconstructions (188).Based on a comparison of the different cost functions derived for themultiple hypothesis reconstructions, the processing circuitry of videodecoder 30 may select one of the hypothesis reconstructions (190). Inthis way, the processing circuitry of video decoder 30 may perform thetemplate-based reconstruction techniques of this disclosure to performhypothesis reconstruction for a current block of encoded video data, byleveraging stored information in the form of previously-reconstructedblocks stored to video data memory 151.

In some instances, each pre-stored residual sample of the identified setof pre-stored residual samples is included in a quantized transformcoefficient block. In some examples, each pre-stored residual sample ofthe identified set of pre-stored residual samples is included in adequantized transform coefficient block. According to someimplementations in which each pre-stored sample is included in adequantized transform coefficient block, video decoder 30 may assign afirst absolute value to a first transform coefficient of each pre-storedresidual sample of the set of pre-stored residual samples, and assign asecond absolute value to all remaining transform coefficients of eachpre-stored residual sample of the set of pre-stored residual samplesexcept for the respective first transform coefficients that are assignedthe first absolute value, the second absolute value being different fromthe first absolute value. In these examples, video decoder 30 may applyan inverse transform to each pre-stored residual sample of the set ofpre-stored residual samples to form a set of dequantized reconstructedresidual samples.

According to some implementations in which each pre-stored sample isincluded in a dequantized transform coefficient block, video decoder 30may scale each dequantized residual sample of the set of dequantizedresidual samples using a dequantized value associated with each of thefirst transform coefficients, and may reconstruct the current blockusing the set of scaled dequantized residual samples. In some examples,one or more pre-stored residual samples of the identified set ofpre-stored residual samples are stored at a fractional precision in thestored portion of the video data. In some examples, video decoder 30 mayclip each pre-stored residual sample of the identified set of pre-storedresidual samples using a pre-defined factor such that each clippedpre-stored residual sample can be represented using four (4) bits orless of data.

In some examples, to combine the set of pre-stored residual samples toform the combination of pre-stored residual samples, video decoder 30may perform a linear combination of the set of pre-stored residualsamples to form a linear combination of pre-stored residual samples. Insome examples, to perform the linear combination of the set ofpre-stored residual samples, video decoder 30 may calculate a weightedsum of the set of pre-stored residual samples, at least in part byassigning a respective weight to each pre-stored residual sample of theset of pre-stored residual samples. In some examples, to combine the setof pre-stored residual samples to form the combination of pre-storedresidual samples, video decoder 30 may merge the set of pre-storedresidual samples to form a single merged residual block, and mayoverwrite the set of pre-stored residual samples with the single mergedresidual block in a memory device.

According to some examples, video decoder 30 may derive a singleresidual block using one or more pre-stored residual samples of theidentified set of pre-stored residual samples, and may overwrite the setof pre-stored residual samples with the derived single residual block ina memory device. In some examples, the stored portion of the video dataincluding the set of pre-stored residual samples is stored to a firstmemory device, and video decoder 30 may load one or more pre-storedresidual samples of the set of pre-stored residual samples from thefirst memory device to a second memory device on one of a once-perpicture basis or a once-per largest coding unit (LCU) basis. The firstmemory device may represent an off-chip memory and the second memorydevice may be an on-chip memory. Again, off-chip memory generallyrepresents a cheaper memory than on-chip memory.

FIG. 7 is a flowchart illustrating an example process 220 by which avideo coding device may perform selective sign prediction techniques ofthis disclosure as part of coding a current block of video data.Although a variety of video coding devices may perform process 220, FIG.7 is described as being performed by video decoder 30 (and morespecifically, processing circuitry thereof), as a non-limiting example.Process 220 may begin when the processing circuitry of video decoder 30determines that a current block of encoded video data is eligible to bedecoded using sign prediction (222). In turn, the processing circuitryof video decoder 30 may determine whether or not the current block meetsa block-based criterion (decision block 224).

In various examples, the processing circuitry of video decoder 30 mayset the block-based criterion based on various factors, including, butnot limited to, the width of the current block, the height of thecurrent block, a coding mode according to which the current block isencoded, or a position of one or more sign-predicted transformcoefficients within the current block. If the processing circuitry ofvideo decoder 30 determines that the current block meets thepredetermined block-based criterion (YES branch of decision block 224),then the processing circuitry of video decoder 30 may perform signprediction with respect to the current block (226). In turn, theprocessing circuitry of video decoder 30 may code (in this example,decode) the current block based on the performed sign prediction (228).However, if the processing circuitry of video decoder 30 determines thatthe current block does not meet the predetermined block-based criterion(NO branch of decision block 224), then the processing circuitry ofvideo decoder 30 may code (in this example, decode) the current blockwithout performing sign prediction (230).

In some examples, the processing circuitry of video decoder 30 maydetermine that the block meets the predetermined criterion only if thewidth of the block is equal to or greater than a predetermined thresholdwidth. In some examples, the processing circuitry of video decoder 30may determine that the block meets the predetermined criterion only ifthe height of the block is equal to or greater than a predeterminedthreshold height. In some examples, the processing circuitry of videodecoder 30 may determine that the block meets the predeterminedcriterion only if the sign-predicted transform coefficients arepositioned in a top-left sub-block of the block.

According to some implementations in which the criterion is based on thetransform coefficients positioned in the top-left sub-block, thetop-left sub-block is a square sub-block. In various examples, thesquare sub-block may have a dimensionality of one of 2×2, 3×3, 4×4, or8×8. In some instances of the square sub-block having the 2×2dimensionality, the processing circuitry of video decoder 30 may performthe sign prediction using a total of four possible predicted signs.

In some examples, to code the video data, the processing circuitry ofvideo decoder 30 may decode an encoded block of the video data. In somesuch examples, the processing circuitry of video decoder 30 may use afirst inverse transform to perform a hypothesis reconstruction processwith respect to the encoded block, select the sign prediction from aplurality of sign predictions based on the hypothesis reconstructionprocess, and reconstruct the encoded block using a second inversetransform that is different from the first inverse transform. In somesuch examples, the processing circuitry of video decoder 30 may receivedata representative of the predetermined criterion in an encoded videobitstream. In these examples, video encoder 22 may signal the datarepresentative of the predetermined criterion in the encoded videobitstream.

FIG. 8 is a flowchart illustrating an example process 250 by which avideo coding device may perform cost function measurement techniques ofthis disclosure as part of a hypothesis reconstruction for a currentblock. Although a variety of video coding devices may perform process250, FIG. 8 is described as being performed by video decoder 30 (andmore specifically, by processing circuitry thereof), as a non-limitingexample. Process 250 may begin when video decoder 30 obtains multiplehypothesis reconstructions for a current block, by applying differentsign combinations to transform coefficients of the current block (252).In an example use case scenario where video decoder 30 performshypothesis reconstruction using two signs (0 and 1, for example), videodecoder 30 may use a total of four sign combinations ({0,0}, {0,1},{1,1}, and {1,0}) to obtain the multiple hypothesis reconstructions forthe current block.

In turn, video decoder 30 may measure respective cost functions for themultiple hypothesis reconstructions using pixel value gradients betweenthe samples of the hypothesis reconstructions and samples of one or moreneighboring blocks (254). For instance, video decoder 30 may measure therespective cost functions by calculating second-order differenceinformation (e.g. differential or derivative data) between absolutevalues of border samples of the hypothesis reconstructions and absolutevalues of the samples of the neighboring block(s). Video decoder 30 maycompare the cost functions measured for the multiple hypothesisreconstructions against one another (256).

Based on the comparison of the different cost functions measured for themultiple hypothesis reconstructions, video decoder 30 may select one ofthe hypothesis reconstructions to use for decoding the current block(258). In this way, video decoder 30 may reduce redundancies, byimplementing gradient-based cost measurement techniques of thisdisclosure.

In some examples, to determine the first set of pixel value gradients,the processing circuitry of video decoder 30 may determine second-orderdifferences between absolute values of the samples of the firsthypothesis reconstruction of the block and absolute values of thesamples of the neighboring block. In these examples, to determine thesecond set of pixel value gradients, the processing circuitry of videodecoder 30 may determine second-order differences between absolutevalues of the samples of the second hypothesis reconstruction of theblock and absolute values of the samples of the neighboring block.

In some examples, the samples of the first hypothesis reconstruction ofthe block used for measuring the first cost function are positionedalong a boundary between the first hypothesis reconstruction of theblock and the neighboring block. In these examples, the samples of thesecond hypothesis reconstruction of the block used for measuring thesecond cost function are positioned along a boundary between the secondhypothesis reconstruction of the block and the neighboring block. Insome examples, the processing circuitry of video decoder 30 may apply athird sign combination to the transform coefficients of the block of thevideo data to obtain a third hypothesis reconstruction with respect tothe block of the video data, and apply a fourth sign combination to thetransform coefficients of the block of the video data to obtain a fourthhypothesis reconstruction with respect to the block of the video data.

In some such examples, the processing circuitry of video decoder 30 mayuse a Gray code order to search among the first hypothesisreconstruction, the second hypothesis reconstruction, the thirdhypothesis reconstruction, and the fourth hypothesis reconstruction. Touse the Gray code to search, the processing circuitry of video decoder30 may search according to the following order of sign combinations: {0,0}, {0, 1}, {1, 1}, {1, 0}.

Aspects of this disclosure are discussed in bullet and sub-bullet formbelow. It will be appreciated, however, that aspects listed in thevarious bullets and sub-bullets below may be applied individually, or inany combination, in accordance with the techniques of this disclosure.

The bullets and sub-bullets listed below are generally directed totechniques of this disclosure that are applicable to a pre-stored set ofreconstructed residual samples. According to these aspects of thedisclosure, a set of fully or partially reconstructed residual blocks,each pre-calculated using a pre-defined transform coefficient block(explained in further detail below), are pre-stored for both videoencoder 22 and video decoder 30. When doing a hypothesis reconstruction,a selected group of reconstructed residual blocks are identified fromthe set of stored reconstructed residual blocks, and the hypothesisreconstruction is generated or estimated using a combination, forexample linear, of these identified stored reconstructed residualblocks.

-   -   a) Each of the stored set of reconstructed residual blocks may        be a reconstructed block using a pre-defined quantized transform        coefficient block, a given dequantization parameter, a given        primary transform and/or a given secondary transform.    -   For example, for a particular coefficient f(x,y) in an M×N TB,        where 0<=x<M, 0<=y<N, given the inverse transform method        (including primary transform and/or secondary transform) and        dequantization parameter, a quantized coefficient block B is        generated by setting f(x,y) to a certain value, for example one        (1), and all other coefficients are set to another value, for        example zero (0), then a fully or partially reconstructed        residual block R is pre-calculated by applying the given        dequantization and inverse transform on B. The reconstructed        residual block R is pre-stored in memory. In this way, during        sign prediction, if the sign of coefficient f(x,y) needs to be        predicted in an M×N TB with the same inverse transform and        dequantization parameter, the techniques of this disclosure        enable video encoder 22 and/or video decoder 30 to avoid doing        inverse transform for f(x,y). Instead, the pre-stored        reconstructed residual block R may be scaled by a factor of        |f(x,y)| and then is used to estimate the reconstructed block.    -   b) Each of the set of stored reconstructed residual blocks may        be the reconstructed block using a pre-defined dequantized        transform coefficient block, a given primary transform and/or a        given secondary transform.        -   For example, for a particular coefficient f(x,y) in an M×N            TB, where 0<=x<M, 0<=y<N, given the inverse transform method            (including primary transform and/or secondary transform), a            dequantized coefficient block B is generated by setting            f(x,y) to a certain value, for example one (1), and all            other coefficients are set to another value, for example            zero (0), then a fully or partially reconstructed residual            block R is pre-calculated by applying the given inverse            transform on B. The reconstructed residual block R is            pre-stored in memory. In this way, during sign prediction,            if the sign of quantized coefficient f(x,y) needs to be            predicted in an M×N TB with the same inverse transform, the            techniques of this disclosure enable video encoder 22 and/or            video decoder 30 to avoid doing inverse transform for            f(x,y). Instead, the quantized coefficient f(x,y) is first            dequantized to be f(x,y)′, then the pre-stored reconstructed            residual block R may be scaled by a factor of |f(x,y)′| and            then is used to estimate the reconstructed block.    -   c) The set of pre-stored reconstructed residual blocks may be        stored with fractional precision, or equivalently, the set of        reconstructed residual blocks may be stored by first scaling        with a pre-defined factor, i.e., 2, 4, 8, 16, and rounded to the        nearest integer. In this way, the precision of the pre-stored        reconstructed residual blocks is increased.    -   d) The set of pre-stored reconstructed residual blocks may be        stored with Clipped value to fit the restricted bit range of        representation. For example, by applying Clipping with a minimum        value Vmin=−8 and maximum value Vmax=7, the entries of all        pre-stored reconstructed residual values can be represented in 4        bits. Bullets (c) and (d) can be applied jointly or        individually, in addition with various other combinations of the        techniques described in this disclosure.    -   e) In one example, a linear combination of these identified        stored reconstructed residual blocks is a weighted sum of these        identified stored reconstructed residual blocks, and the weights        may be decided by the level value of a quantized or dequantized        transform coefficient for which the sign is being predicted.    -   f) Because different transform and/or quantization methods may        be available for video encoder 22 and video decoder 30,        multiples of reconstructed blocks may need to be pre-stored for        one particular quantized coefficient block, which may increase        the burden of storing and accessing the pre-calculated residual        blocks. To handle this issue, a merging process may be performed        on the pre-stored residual blocks according to aspects of this        disclosure, such that multiple residual blocks are merged and        represented by one residual block.        -   Alternatively, a function ƒ is pre-defined to derive one            pre-stored residual block from another pre-stored residual            block, such that only one of several residual blocks may            need to be stored and others are derived using the            pre-defined function ƒ    -   g) The total size of the set of pre-calculated residual blocks        may be large, and may be difficult to be stored in on-chip        memory. Alternatively, the set of pre-calculated residual blocks        is pre-stored in external (e.g., cheaper) memory, and only the        needed residual blocks are loaded to on-chip memory once per a        bigger block or picture area, for example largest coding unit        (LCU).    -   h) In one example, each of the pre-stored set of partial        residual blocks is a template (e.g., only the leftmost and        topmost residual pixels are stored) for sign prediction. The        template based sign prediction process described in Henry can be        applied in conjunction with the pre-stored set of partial        residual blocks in a way that the template is not generated by        doing partial inverse transform on the fly. Instead, the        template is generated by accessing the pre-stored set of partial        residual blocks and performing a scaling operation.

The bullets and sub-bullets listed below are generally directed totechniques of this disclosure that generally relate to conditions ofusing sign prediction. According to these aspects of this disclosure,several conditions may be applied for blocks which can apply signprediction. These conditions may include, but are not limited to, theblock width and height, block area size, transform coefficient frequencylocations, last position, intra prediction mode, whether only primarytransform is used to code a block, whether only particular transformfunctions or certain transform subset are used, inverse transformmethod, whether it is intra or inter coded blocks, or any other decodedinformation of the current and/or neighbor block(s).

-   -   a. In one example, sign prediction is not applied if either of        block width or block height is larger than a threshold, where        the threshold may be pre-defined. (Where otherwise, the sign        prediction would be applied according to existing sign        prediction technology). Alternatively, the threshold can be        configurable and may be signaled. A typical threshold value can        be, but is not limited to 4, 8, 16, 32, or 64.        -   Alternatively, the threshold value may be dependent on coded            modes. In one example, the threshold may be different for            intra and inter coded blocks. For example, for an intra            coded block, the threshold value may be 32, while for an            inter coded block, the threshold value may be 64.        -   Alternatively, sign prediction is not applied if the minimum            or maximum of block width or block height is larger than a            threshold.    -   b. In another example, sign prediction is not applied for        transform coefficients which are not located in the top-left K×J        part of a TB, where a typical setting of the K×J value can be        2×2, 3×3, 4×4 or 8×8, etc. (Where otherwise, the sign prediction        would be applied according to existing sign prediction        technology). Alternatively, the values of K and/or L may further        depend on coded modes, and/or quantization parameters.    -   c. The inverse transform performed in the sign prediction        process, for example to estimate the hypothesis, may be limited        to certain inverse transform(s). The selected transform(s) can        be different from the inverse transform to which is applied to        generate the actual reconstructed block after the actual        coefficient signs have been derived.        -   In one example, only discrete cosine transform to type-II            (DCT-II) is used as the primary inverse transform during            sign prediction.        -   In another example, during sign prediction, secondary            transform is replaced by transform using pre-defined            permutation matrix, and the pre-defined permutation matrix            may include entities of “+1” and “−1”.        -   In another example, secondary transform is not used during            sign prediction.    -   d. In one example, sign prediction may be applied only for the        blocks that use only primary transform. In this example, the        complexity of the inverse transform may be smaller than the        complexity arising from having a primary and a secondary        transform applied together. In another example, sign prediction        may be applied only for the blocks that use certain transform        core or certain transform set/subset, because it is possible        that sign prediction may not work efficiently for all transform        basis/cores, and complexity can be reduced by limiting the sign        prediction application.

The bullets and sub-bullets listed below are generally directed totechniques of this disclosure that generally relate to cost functions.According to these aspects of this disclosure, the cost function formeasuring each hypothesis reconstruction may utilize other decodedinformation, including, but not limited to, the block width and height,block area size, transform coefficient frequency locations, lastposition, intra prediction mode, whether blocks are intra or inter codedblocks, or any decoded information of the current and/or neighborblock(s). In various examples, video encoder 22 and/or video decoder 30(or one or more components thereof) may implement the cost functionaspects of this disclosure according to one or more of the techniqueslisted below.

-   -   a. The cost function may be configured to measure the pixel        value difference or pixel gradient value difference along the        intra prediction direction.        -   a. In one example, a partial subset of allowed intra            prediction directions are pre-selected, and each allowed            intra prediction direction is mapped to one of the            pre-selected intra directions which approximates the intra            prediction direction. In addition, the pixel value            difference along the mapped direction is measured in the            cost function.        -   b. In case the intra prediction direction is not available            (e.g., one block is coded with inter mode), a derived intra            prediction mode may be used instead. The derived intra            prediction mode may come from the spatial neighboring coded            blocks, or a block identified by a motion vector in a coded            picture.        -   c. When the intra prediction direction points to a            fractional position, an interpolation process is applied to            derive the value of that fractional position. The            interpolation filter used for generating intra prediction            block may be applied, or a different and/or simpler            interpolation filter is used.    -   b. The cost function may be measured by the second order or N-th        order of pixel value difference.        -   a. Multiple reference lines (e.g., more than 2) are used to            measure the cost value.        -   b. The number of reference lines used for measuring the cost            value may depend on coded information, including but not            limited to intra prediction mode, neighboring reconstructed            values.        -   c. The maximum number of reference lines may be set equal to            the required number of reference lines for intra prediction.    -   c. Some pixels are excluded from measuring the cost value        depending on a given criterion.        -   a. In one example, the criterion includes whether the            absolute value of 1-st order, 2-nd order (gradient), or N-th            order derivative is less than a given threshold value.        -   b. In another example, the criterion is whether the            neighboring reconstructed samples used to calculate the N-th            order derivative is available.        -   c. In another example, the criterion may include whether            identity transform/transform skip is applied to the            horizontal/vertical transform.    -   d. When measuring the cost using neighboring reconstructed        values, instead of calculating the 1-st order, 2-nd order        (gradient) or N-th order differences along a fixed or        pre-defined direction (e.g., for left block boundary it is        horizontal direction, for top block boundary, it is vertical        direction), the cost values for multiple directions are        calculated, and the minimum cost is used for measuring the cost.    -   e. For a larger block size, the cost function may use a        downsampled hypothesis reconstruction, so that the complexity of        applying cost function may be reduced for larger block sizes.    -   f. Pixel value difference may be measured only for partial        pixels in the reconstructed and neighbor blocks. Those pixels        can be, for example, corners, where typically error is larger,        or those corner pixels can be excluded from the difference        measure as potential outliers, or different weights can be        assigned for those selected pixels.    -   g. Instead of measuring the 1st order, gradient (2nd order)        difference between the neighboring reconstructed samples and the        reconstructed samples located at the current block boundary,        estimated boundary residual samples are generated by subtracting        neighboring reconstructed samples and the prediction samples        located at the current block boundary. Then the difference        between the estimated boundary residual samples and the        candidate boundary residual samples are measured. In this way,        techniques of this disclosure can avoid a duplicate        reconstruction process which needs to add the same prediction        samples and different candidate residual samples. As used        herein, the candidate residual samples refer to the residual        samples generated using one candidate signs combination.    -   h. The number of predicted signs (e.g., 0, 1, 2, 3, 4, 5, 6) may        depend on decoded/encoded information, including but not limited        to intra prediction mode, block width/height, block size, QP,        color components, temporal layer, intra/inter coded, whether        transform skip/identity transform is applied, whether        non-separable secondary transform is applied, quantized        coefficients, EMT index, PDPC flag/index.    -   i. When multiple ‘M’ signs are to be predicted, instead of        testing all the possible 2^(M) hypothesis reconstructions, the        ‘M’ signs are divided into ‘N’ groups, then the sign prediction        is performed in unit of group. Each group of signs are predicted        by setting the remaining groups of signs as previously-predicted        values or default value if they have not been previously        predicted yet. This process can iterate several rounds until a        pre-defined limitation is met.        -   The group of signs with larger transform coefficient levels            are predicted prior to the group of signs with relatively            lower coefficient levels.        -   The iteration may be terminated when the predicted signs for            each group no longer change or the iteration number has met            a pre-defined threshold value, or the cost difference            between iterations (for example, succeeding iterations) is            not exceeding a certain threshold.

The bullets and sub-bullets listed below are generally directed totechniques of this disclosure that generally relate to the selection ofwhich coefficient signs will be predicted. According to these aspects ofthis disclosure, the selection of predicted transform coefficients mayutilize other decoded information, including, but not limited to, theblock width and height, block area size, transform coefficient frequencylocations, last position, intra prediction mode, whether sign datahiding is applied, whether blocks are intra or inter coded blocks, orany decoded information of the current and/or neighbor block(s). Invarious examples, video encoder 22 and/or video decoder 30 (or one ormore components thereof) may implement the coefficient sign selectionaspects of this disclosure according to one or more of the techniqueslisted below.

-   -   a. Several threshold values, e.g., t₀, t₁, . . . , t_(N), where,        for example, t₀>t₁> . . . >t_(N), may be pre-defined, and        several intervals are derived as (t₀, +∞), [t₁, t₀), [t_(N),        t_(N-1)) in a descending order. Dequantized coefficient levels        are compared to each one of the given intervals until one        interval which covers the dequantization coefficient level is        identified. The signs of coefficients are identified as        predicted signs according to the descending order. For example,        the signs for coefficients in interval (to, +∞) are the first to        be put into the list of predicted signs, then the signs for        coefficients in interval [t₁, t₀) are put into the list of        predicted signs, and this process continues for each interval in        descending order until the number of predicted signs reached the        pre-defined maximum number of predicted signs, or all available        intervals have been checked. If a coefficient has a level less        than t_(N), the sign may not be considered as a predicted sign.        A typical setting of intervals can be (600, +∞), [300, 600),        which means, for example, the signs of dequantized coefficient        levels being less than 300 are not considered as predicted        signs.        -   a. In one example, the intervals are selected depending on            coded information, including but not limited to the QP,            and/or block size, block width, block height.    -   b. A number of maximum predicted signs may depend on decoded        information, e.g., whether the block is inter or intra coded, or        inverse transform method (e.g., which primary/secondary        transform is applied).    -   c. When sign prediction is applied to a certain TB, sign data        hiding may be still applied. In this case, instead of hiding the        first nonzero coefficient in the forward scanning order, the        sign of the last nonzero coefficient in the forward scanning        order may be hidden. In one example, furthermore, the sign data        hiding may not be performed at the CG-level. That is,        consecutive nonzero coefficients, even if located in different        CGs, may be jointly considered for sign data hiding.

Some aspects of this disclosure are directed to determining the CABACcontext for coding coefficient signs. The context value for codingcoefficient signs may depend on the dequantized level (absolute value)of the corresponding coefficient. Several intervals are pre-defined, oneof the context value is decided by the index of interval which thedequantized level value belongs. A common or even typical setting ofintervals are (0, 600), [600, +∞), or (0, 600], (600, +∞). With thissetting, if the level if the level is in the range of (0, 600), thecontext value is 0, otherwise, the context value is 1. In variousexamples, video encoder 22 and/or video decoder 30 (or one or morecomponents thereof) may implement the CABAC context determinationaspects of this disclosure according to one or more of the techniquesdescribed below.

-   -   a. The intervals depend on coded information, including, but not        limited to the QP, and/or block size, block width, block height.

Some aspects of this disclosure are directed to a searching order forhypothesis. Each hypothesis consists of one possible combination ofcandidate sign values. For example, when 2 signs are predicted, thereare totally 4 hypotheses (4 possible combination of candidate signvalues) including:

Hypothesis index 0: {0, 0}

Hypothesis index 1: {0, 1}

Hypothesis index 2: {1, 0}

Hypothesis index 3: {1, 1},

where the first digit indicates the first predicted sign, and the seconddigit indicates the second predicted sign, 0 indicates positive sign and1 indicates negative sign, and the combination of candidate sign valuesin each hypothesis corresponds to the binarized representation of thehypothesis index. It is noted that if a system searches the hypothesisin the increasing/decreasing order, there can be more than 1 change ofsign values between two neighbor hypotheses, e.g., from hypothesis index1 to index 2, there are two signs changing the values, this means thatwe need to consider the boundary samples change incurred by two signs.

In various examples, video encoder 22 and/or video decoder 30 (or one ormore components thereof) may implement the hypothesis searching orderaspects of this disclosure according to one or more of the techniquesdescribed below. To reduce the complexity, according to aspects of thisdisclosure, the searching order following the Gray code is applied:

Hypothesis index 0: {0, 0}

Hypothesis index 1: {0, 1}

Hypothesis index 2: {1, 1}

Hypothesis index 3: {1, 0},

It can be noted that, by assigning the Gray code value for eachhypothesis, there is only 1 change of sign values between two neighborhypotheses. In this way, based on the generated boundary sample valuesof a previous hypothesis, for current hypothesis, to generate theboundary sample values, systems and techniques of this disclosure onlyconsider the boundary samples change incurred by one sign. This reducesthe complexity and regularize the cost calculation process.

Some aspects of this disclosure are directed to techniques formaximum-likelihood decoder-side estimation of transform coefficientsigns for video coding. In recent video compression standards, likeHEVC, the signs of nonzero transforms coefficients (positive ornegative) are coded using one bit per sign. Several methods had beenproposed to reduce the average bit rate needed to code those signs, butprocesses using direct estimation are relatively complex, and yieldsmall gains. One example method is to estimate the signs indirectly, byhaving the decoder enumerate all combinations of some sign values, applythe inverse transform for each case, and find the one that maximizessmoothness along the borders with known blocks.

This disclosure introduces techniques that extend this approach byemploying a new maximum-likelihood (ML) formulation of the problem,which shows that the previous techniques are in fact empiricapproximations of the ML technique. Using those new tools, thisdisclosure describes several new techniques that may improve signestimation, through better use of statistical properties and techniques.The changes, described in more detail later in this disclosure, aresummarized below:

-   -   Theoretical and practical advantages of using the sum of        absolute difference (SAD), instead of sums of squares;    -   Exploiting information from any set of inverse transform values,        and not only those along block boundaries;    -   Decisions and entropy coding based on sets of weights matched to        each type of block prediction (temporal, spatial, directional,        etc.), and also matched to the type of transform;    -   The use of second order prediction, using estimates of the        variances of errors in pixel predictions, to improve the sign        discrimination functions.

In video compression standards, a linear transformation may be appliedto the video signal before coding, and the nonzero coefficients havetheir magnitude coded first, followed by the sign (positive ornegative). Several techniques have been proposed to use context-basedarithmetic coding to more efficiently encode the magnitudes of transformcoefficients, but the advantages of using context coding for the signshave been constrained mostly to wavelet transforms, and in more recentstudies on the application for video block transforms, it has beendecided that it makes more practical sense to code the signs using onebit, without context coding (corresponding to the bypass mode in theHEVC standard).

There is a technique to avoid, in some cases, coding one sign in a setof coefficients, called “sign data hiding,” (SDH) as set forth in“Transform coefficient coding in HEVC” by J. Sole, R. Joshi, N. Nguyen,T. Ji, M. Karczewicz, G. Clare, F. Henry, and A. Dueñas in IEEE Trans.Circuits and Systems for Video Technology, December 2012, pp. 1765-1777,but such techniques may be unrelated to the techniques of thisdisclosure (i.e., both techniques can be used independently).

In other examples, techniques for guessing the correct sign valuesinclude enumerating all combinations of sign values, applying theinverse transform for each case, and then using information fromneighboring blocks, or statistical models, to select the combinationthat produces the block that best matches some additional informationabout the block being coded.

FIG. 9 shows an example of a system for predicting signs (at a videodecoder, such as video decoder 30) by evaluating boundary pixelsobtained by applying the inverse transform, using arrays of signguesses. The application of this idea to image compression was firstproposed in “Prediction of signs of DCT coefficients in block-basedlossy image compression” by N. Ponomarenko, A. V. Bazhyna, and K. O.Egiazarian, in Proc. SPIE Vol. 6497: Image Process. Algorithms Syst. V,San Jose, Calif., USA, January 2007, and later adapted to video codingin “Coefficient sign bit compression in video coding” by J. Koyama, A.Yamori, K. Kazui, S. Shimada, and A. Nakagawa, in Proc. 29th PictureCoding Symp., Krakow, Poland, May 2012, pp. 385-388, and the basicapproach is shown in FIG. 9. It exploits the fact that the probabilityof the sign being correctly predicted with these enumerative methodsgrows with the magnitude of its corresponding coefficient. For example,if the magnitude of a coefficient is in a certain range of relativelylarge values, then the sign of this coefficient can be predictedcorrectly, on average, in 95% of the cases, and if the coefficientmagnitude is relatively small then it can predicted in only 55% percentof the cases.

It is this difference in probabilities that can be exploited by anarithmetic encoder, to reduce the number of bits to code the signs ofthe transform coefficients with largest magnitudes in a bloc, asproposed by Ponomarenko et al in “Prediction of signs of DCTcoefficients in block-based lossy image compression” cited above. Thisdisclosure may address one or more problems with existing techniques forsign estimation. It has been shown that the sign estimation method ofFIG. 9 can effectively improve compression, but it is based purely onempirical observations, which are not very useful for designing improvedversions, especially in the latest versions of video codecs, whichemploy many different prediction modes, and residual transforms. Forexample, if in an effort to improve its efficacy, this disclosureintroduces techniques that extend the set of pixels to be tested to thewhole block, and use the same prediction that has been used by theencoder, the method will not work.

This happens because the transform is orthogonal, and thus the sum ofsquares of the possible residual values is independent of the sign ofthe transform coefficients, and will remain constant for all sets ofsigns that are tested. When considered that a substantial amount ofcomputation is employed by an encoder to find a very good prediction forthe whole block, it is paradoxical that it cannot be used to helpestimating the transform coefficient signs. The description belowexplains that the paradox is not intrinsically related to the estimationproblem, but instead it is a consequence of using a suboptimalestimation method.

Thus, the main problem solved by the techniques of this disclosure ishow to optimize the technique of estimating transform coefficient signsat the decoder, using advanced statistical methods (instead of ad hoctechniques) for all the different type of prediction modes, andtransform types.

The techniques of this disclosure will now be described, starting withan overview of the general approach used. The techniques of thisdisclosure may solve the problems discussed above considering that,instead of just employing empiric best match criteria to decide the signvalues, the techniques of this disclosure consider that this problem canbe formulated as a statistical sign estimation problem, and thattherefore can be solved with advanced statistical methods.

Thus, the techniques of this disclosure use the maximum-likelihoodapproach, and modify the system of FIG. 9 to the one system shown inFIG. 10, where the new elements are marked as “transform coefficients,”coefficient weights,” “select largest factors,” “coefficient indexes,”“sign arrays,” “recover block pixels,” “block subset,” “modelparameters,” log-likelihood computation,” “predicted signs,” “signcorrections,” and “decoded pixels.” The following portion of thisdisclosure discusses how each of those stages may potentially beimproved. This section summarizes, in general terms, how the maximumlikelihood technique is applied to the sign estimation problem.

In a block of dimension W×H, symbol s indicates an array of signs thatare being tested, use

c _(s) [x,y], x=0,1, . . . ,W−1; y=0,1, . . . ,H−1,

to indicate the transform coefficients with sign modified according tos, and represent the transformed values as

v _(s) [x,y], x=0,1, . . . ,W−1; y=0,1, . . . ,H−1.

In addition, define K to represent the knowledge that the decoder hasavailable to estimate the signs (e.g., decoded pixels in neighboringblocks, transform coefficients with known signs, statistical models,etc.).

FIG. 10 shows an example of a system for predicting signs (at a videodecoder such as video decoder 30) using a maximum-likelihood accordingto the techniques of this disclosure.

Looking for array s that, given K, maximizes the conditional likelihoodfunction

${{\Lambda_{K}(s)} = {\prod\limits_{x = 0}^{W - 1}\; {\prod\limits_{y = 0}^{H - 1}\; {p\left( {{v_{s}\left\lbrack {x,y} \right\rbrack}K} \right)}}}},$

where p(v_(s) [x, y]|K) is the probability density function of theresiduals, given decoder knowledge K. Equivalently, the log-likelihoodfunction can be maximized as follows:

L _(K)(s)=−log(Λ_(K)(s)).

The probability distribution functions are commonly defined according tosome model for the data. For example, in imaging applications it isknown that residual values have probabilities fairly well represented bythe Generalized Gaussian Distributions, with probability densityfunction

${{p(v)} = {\frac{\zeta \; r}{2\; \sigma \; {\Gamma \left( {1/r} \right)}}{\exp \left( {- {{{\zeta \left\lbrack {v - \mu} \right\rbrack}/\sigma}}^{r}} \right)}}},$

where r, μ, σ represent a shape parameter, the mean, and standarddeviation, respectively, and

$\zeta = {\sqrt{\frac{\Gamma \left( {3/r} \right)}{\Gamma \left( {1/r} \right)}}.}$

Thus, for the sign estimation problem, to evaluate an array of signvalues s, a function that is equal to the log-likelihood, except forconstant terms, in the form

$\begin{matrix}{{\Phi_{K}(s)} = {\sum\limits_{x = 0}^{W - 1}\; {\sum\limits_{y = 0}^{H - 1}\; {\frac{{v_{s}\left\lbrack {x,y} \right\rbrack} - {\mu_{K}\left\lbrack {x,y} \right\rbrack}}{\sigma_{K}\left\lbrack {x,y} \right\rbrack}}^{r_{k}}}}} & (1)\end{matrix}$

After these definitions, how the decoder-side sign estimation techniquecan be integrated into a complex video coding system can be defined.

-   -   1. For each prediction mode and transform type, determine the        knowledge that is available after decoding the transform        coefficient magnitudes, and define a conditional statistical        model for the residuals.    -   2. Given the statistical models, determine its typical        parameters (e.g., from training), and using those parameters        design an optimized sign estimation method, that has specific        rules for each of the prediction modes, and transform types.    -   3. Integrate the sign estimation enumeration into the encoding        and decoding programs, adding the methods and data to determine        the optimization parameters and criteria, which are specific for        each prediction mode and transform type.

For example, when using eq. (1), the conditional statistical model iscompletely defined by the shape, mean, and variance parameters, i.e.,r_(K), μ_(K)[x, y], σ_(K)[x, y], which should be also be estimatedduring encoding and decoding, but with much reduced accuracy, orsimplified models. For example, the following criteria may be used:

-   -   If the residual distributions are assumed to be Laplacian (shape        parameter r=1), then the following can be used:

${\Phi_{K}(s)} = {\sum\limits_{x = 0}^{W - 1}\; {\sum\limits_{y = 0}^{H - 1}{{{w_{K}\left\lbrack {x,y} \right\rbrack}\left( {{v_{s}\left\lbrack {x,y} \right\rbrack} - {\mu_{K}\left\lbrack {x,y} \right\rbrack}} \right)}}}}$

with an array of positional weights

${w_{K}\left\lbrack {x,y} \right\rbrack} \approx {\frac{1}{\sigma_{K}\left\lbrack {x,y} \right\rbrack}.}$

This means that in this case the sum of absolute differences (SAD), orthe weighted sum of absolute differences (WSAD) should be used formeasuring how a set of recovered pixels fits the statistical model.

-   -   If the residual distributions are assumed to be Gaussian (shape        parameter r=2), then the following can be used:

${\Phi_{K}(s)} = {\sum\limits_{x = 0}^{W - 1}\; {\sum\limits_{y = 0}^{H - 1}\left\lbrack {{w_{K}\left\lbrack {x,y} \right\rbrack}\left( {{v_{s}\left\lbrack {x,y} \right\rbrack} - {\mu_{K}\left\lbrack {x,y} \right\rbrack}} \right)} \right\rbrack^{2}}}$

In this case the sum of square differences (SSD), or the weighted sum ofsquare differences (WSSD) should be used for measuring the fit to themodel. Note that various existing technologies can be considered as anapproximation of this last equation, where weights are equal to one (1)if the residual is in the block boundary, and equal to zero otherwise,and the estimated means are equal to the difference between currentprediction and pixel values from neighboring blocks. The next sectionsof this disclosure present some of the details on how the processdescribed above is used, and the ways it is different from existingtechnologies.

Aspects of this disclosure related to variable weights and spatialsupport in the log-likelihood estimation will now be described. Theprevious methods for sign estimation use only pixels along blockboundaries to measure discontinuities created by incorrect sign values.This disclosure introduces techniques for using the log-likelihoodfunction to determine which pixels should be included in the signdecision. As indicated by the log-likelihood objective function of eq.(1), the pixels that have smallest estimated variances have moreimportance in the log-likelihood function. Thus, to reduce thecomplexity of the estimation, this disclosure describes techniques forusing only the pixel values that are expected to have sufficiently smallvariance. This depends on statistical model, prediction method,transform type, etc., and can be different from pixels at blockboundaries.

FIG. 11 shows examples of how this can be done. In the figure the grayvalues are used to indicate the weights used in the log-likelihood, withdarker indicating larger, and white indicating zeros equal to zero (andthus, ignored pixels). Existing techniques may use equal weights inboundary pixels, while the techniques of this disclosure includeallowing for variable in different positions, depending on thestatistics.

FIG. 11 shows examples of how, according to the techniques of thisdisclosure, variable weights in a block (darker represents largerweights, and white represents weights equal to zero) may exist,depending on the statistical model and complexity considerations.

Aspects of this disclosure related to the determination of likelihoodweights from prediction will now be described. One more importantdifference between existing technologies and the use of variableweights, as described in the last section, is that it also allows fortwo different modes for using those weights:

-   -   The set of weights can be pre-defined for different prediction        modes and transform types, and stored in fixed tables.    -   The set of weights can be compared while encoding and decoding,        using an algorithm shared by video encoder 22 and video decoder        30 to estimate statistical parameters, according to the        knowledge available to the decoder (K).

FIG. 12 shows an example of how the encoder and decoder, after analyzingthe properties of the block prediction, decide on the set of weights tobe used for likelihood computation.

FIG. 12 shows the new process: the block prediction, which is shared bythe encoder and decoder, can be used to estimate the variance ofresiduals, and from those variances the weights to be used in thelikelihood function can be derived.

Additional aspects of this disclosure related to transform coefficientselection will now be described. In existing technologies, the set ofcoefficients that need to have their signs estimated by the decoder isdefined solely by the coefficient magnitudes, with the selection of thelargest non-zero coefficients.

The effectiveness of estimating the sign of a coefficient does depend onthe magnitude, but it also depends on other factors, including:

-   -   The set of pixels used for computing the log-likelihood        function, and their variances;    -   The projection of the transform basis functions corresponding to        a coefficient on the set of pixels being evaluated.

The combination of these factors may be important because it determineshow well the estimation will work. For example, if some transformcoefficient has little effect on the pixels where the likelihoodfunction is being evaluated, or it affects only positions where thevariance is very large, then the magnitude of that coefficient may beless important for the sign estimation.

Using this property, this disclosure introduces techniques for changingthe criteria to select the transform coefficients that have their signestimated at the decoder with the following.

-   -   1. For each prediction mode and transform, define a set of        non-negative multipliers

m _(K) [x,y], x=0,1, . . . ,W−1; y=0,1, . . . ,H−1,

with magnitude proportional to an estimate of the effect of thatcoefficient in the likelihood estimation.

-   -   2. Select for sign testing the coefficients with largest product        of magnitude and multipliers, i.e., instead of only the        magnitudes, use the products

m _(K) [x,y]|c[x,y]|, x=0,1, . . . ,W−1; y=0,1, . . . ,H−1,

As in the case of weights for the likelihood function, these multiplierscan be pre-computed, or derived by the encoder and decoder, based onshared prediction data.

Aspects of this disclosure related to entropy coding will now bedescribed. According to existing technologies, the sign correction bitsare entropy coded using contexts that depend on the magnitude of itscoefficient. As indicated in the previous sections, the effectiveness ofthe estimation process depends on the likelihood evaluation, which inturn depends on several other factors.

To potentially improve compression, this disclosure describes techniquesfor adding additional contexts to the entropy coding stage, includingthe following:

-   -   1. Contexts depending on block size and prediction type, i.e.,        intra-frame or inter-frame prediction.    -   2. For intra-frame prediction, adding contexts based on the        prediction mode, or some aggregated data based on prediction        mode.    -   3. Adding additional context based on the transform type. More        specifically, in the JEM-3.0 codec, the transform applies to a        block according to encoded information, according to the        Embedded Multiple Transform (EMT) flag, and the Non-separable        Secondary Transform (NSST) flags. Thus, to be adaptive to the        transform, the context can depend on the EMT and NSST flags, or        some aggregated data based on those flags.

Using contexts defined by intervals of the product of the transformcoefficient magnitude, and its corresponding multiplier, as defined inthe previous section.

Certain aspects of this disclosure have been described with respect toextensions of the HEVC standard for purposes of illustration. However,the techniques described in this disclosure may be useful for othervideo coding processes, including other standard or proprietary videocoding processes not yet developed.

A video coder, as described in this disclosure, may refer to a videoencoder or a video decoder (e.g., a video encoding device or a videodecoding device). Similarly, a video coding unit may refer to a videoencoder or a video decoder. Likewise, video coding may refer to videoencoding or video decoding, as applicable.

It is to be recognized that depending on the example, certain acts orevents of any of the techniques described herein can be performed in adifferent sequence, may be added, merged, or left out altogether (e.g.,not all described acts or events are necessary for the practice of thetechniques). Moreover, in certain examples, acts or events may beperformed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors, rather than sequentially.

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored on or transmitted over as oneor more instructions or code on a computer-readable medium and executedby a hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media, or communication media including any mediumthat facilitates transfer of a computer program from one place toanother, e.g., according to a communication protocol. In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processors to retrieve instructions, codeand/or data structures for implementation of the techniques described inthis disclosure. A computer program product may include acomputer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. Also, any connection is properly termed acomputer-readable medium. For example, if instructions are transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, digital subscriber line (DSL), orwireless technologies such as infrared, radio, and microwave, then thecoaxial cable, fiber optic cable, twisted pair, DSL, or wirelesstechnologies such as infrared, radio, and microwave are included in thedefinition of medium. It should be understood, however, thatcomputer-readable storage media and data storage media do not includeconnections, carrier waves, signals, or other transitory media, but areinstead directed to non-transitory, tangible storage media. Disk anddisc, as used herein, includes compact disc (CD), laser disc, opticaldisc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wheredisks usually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,processing circuitry (e.g., fixed function circuitry, programmableprocessing circuitry, or any combination of fixed function circuitry andprogrammable processing circuitry), application specific integratedcircuits (ASICs), field programmable logic arrays (FPGAs), or otherequivalent integrated or discrete logic circuitry. Accordingly, the term“processor,” as used herein may refer to any of the foregoing structureor any other structure suitable for implementation of the techniquesdescribed herein. In addition, in some aspects, the functionalitydescribed herein may be provided within dedicated hardware and/orsoftware modules configured for encoding and decoding, or incorporatedin a combined codec. Also, the techniques could be fully implemented inone or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, an integratedcircuit (IC) or a set of ICs (e.g., a chip set). Various components,modules, or units are described in this disclosure to emphasizefunctional aspects of devices configured to perform the disclosedtechniques, but do not necessarily require realization by differenthardware units. Rather, as described above, various units may becombined in a codec hardware unit or provided by a collection ofinteroperative hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

Various examples have been described. These and other examples arewithin the scope of the following claims.

What is claimed is:
 1. A method of coding video data, the methodcomprising: identifying a set of pre-stored residual samples in apre-stored portion of the video data; combining the set of pre-storedresidual samples to form a combination of pre-stored residual samples;applying a first sign combination to transform coefficients of thecombination of pre-stored residual samples to obtain a first hypothesisreconstruction with respect to a current block of the video data;applying a second sign combination to the transform coefficients of thecombination of pre-stored residual samples to obtain a second hypothesisreconstruction with respect to the current block of the video data;deriving respective cost functions with respect to the first hypothesisreconstruction and the second hypothesis reconstruction; comparing therespective derived cost functions; selecting, based on the comparison,either the first hypothesis reconstruction or the second hypothesisreconstruction; and coding the current block using a sign predictionassociated with the selected one of the first hypothesis reconstructionor the second hypothesis reconstruction.
 2. The method of claim 1,wherein each pre-stored residual sample of the identified set ofpre-stored residual samples comprises a quantized transform coefficientblock.
 3. The method of claim 1, wherein each pre-stored residual sampleof the identified set of pre-stored residual samples comprises adequantized transform coefficient block.
 4. The method of claim 3,further comprising: assigning a first absolute value to a firsttransform coefficient of each pre-stored residual sample of the set ofpre-stored residual samples; assigning a second absolute value to allremaining transform coefficients of each pre-stored residual sample ofthe set of pre-stored residual samples except for the respective firsttransform coefficients that are assigned the first absolute value, thesecond absolute value being different from the first absolute value;applying an inverse transform to each pre-stored residual sample of theset of pre-stored residual samples to form a set of dequantizedreconstructed residual samples.
 5. The method of claim 4, wherein thefirst absolute value is one (1), and wherein the second absolute valueis zero (0).
 6. The method of claim 4, further comprising: scaling eachdequantized residual sample of the set of dequantized residual samplesusing a dequantized value associated with each of the first transformcoefficients; and reconstructing the current block using the set ofscaled dequantized residual samples.
 7. The method of claim 1, whereinone or more pre-stored residual samples of the identified set ofpre-stored residual samples are stored at a fractional precision in thestored portion of the video data.
 8. The method of claim 1, furthercomprising scaling each pre-stored residual sample of the identified setof pre-stored residual samples using a pre-defined factor.
 9. The methodof claim 1, further comprising clipping each pre-stored residual sampleof the identified set of pre-stored residual samples using a pre-definedfactor such that each clipped pre-stored residual sample can berepresented using four (4) bits or less of data.
 10. The method of claim1, wherein combining the set of pre-stored residual samples to form thecombination of pre-stored residual samples comprises performing a linearcombination of the set of pre-stored residual samples to form a linearcombination of pre-stored residual samples.
 11. The method of claim 10,wherein performing the linear combination of the set of pre-storedresidual samples comprises calculating a weighted sum of the set ofpre-stored residual samples, at least in part by assigning a respectiveweight to each pre-stored residual sample of the set of pre-storedresidual samples.
 12. The method of claim 1, wherein combining the setof pre-stored residual samples to form the combination of pre-storedresidual samples comprises merging the set of pre-stored residualsamples to form a single merged residual block, the method furthercomprising: overwriting the set of pre-stored residual samples with thesingle merged residual block in a memory device.
 13. The method of claim1, further comprising: deriving a single residual block using one ormore pre-stored residual samples of the identified set of pre-storedresidual samples overwriting the set of pre-stored residual samples withthe derived single residual block in a memory device.
 14. The method ofclaim 1, wherein the stored portion of the video data including the setof pre-stored residual samples is stored to a first memory device, themethod further comprising: loading one or more pre-stored residualsamples of the set of pre-stored residual samples from the first memorydevice to a second memory device on one of a once-per picture basis or aonce-per largest coding unit (LCU) basis.
 15. The method of claim 1,wherein coding the video data comprises decoding encoded video data,wherein the current block is an encoded block of the encoded video data,and wherein coding the current block comprises reconstructing theencoded block.
 16. A device for coding video data, the devicecomprising: a video data memory; and processing circuitry coupled to thevideo data memory, the processing circuitry being configured to:identify a set of pre-stored residual samples in a portion of the videodata that is pre-stored to the video data memory; combine the set ofpre-stored residual samples to form a combination of pre-stored residualsamples; apply a first sign combination to transform coefficients of thecombination of pre-stored residual samples to obtain a first hypothesisreconstruction with respect to a current block of the video data; applya second sign combination to the transform coefficients of thecombination of pre-stored residual samples to obtain a second hypothesisreconstruction with respect to the current block of the video data;derive respective cost functions with respect to the first hypothesisreconstruction and the second hypothesis reconstruction; compare therespective derived cost functions; select, based on the comparison,either the first hypothesis reconstruction or the second hypothesisreconstruction; and code the current block using a sign predictionassociated with the selected one of the first hypothesis reconstructionor the second hypothesis reconstruction.
 17. The device of claim 16,wherein each pre-stored residual sample of the identified set ofpre-stored residual samples is included in a quantized transformcoefficient block.
 18. The device of claim 16, wherein each pre-storedresidual sample of the identified set of pre-stored residual samplescomprises a dequantized transform coefficient block.
 19. The device ofclaim 18, wherein the processing circuitry is further configured to:assign a first absolute value to a first transform coefficient of eachpre-stored residual sample of the set of pre-stored residual samples;assign a second absolute value to all remaining transform coefficientsof each pre-stored residual sample of the set of pre-stored residualsamples except for the respective first transform coefficients that areassigned the first absolute value, the second absolute value beingdifferent from the first absolute value; apply an inverse transform toeach pre-stored residual sample of the set of pre-stored residualsamples to form a set of dequantized reconstructed residual samples. 20.The device of claim 19, wherein the first absolute value is one (1), andwherein the second absolute value is zero (0).
 21. The device of claim19, wherein the processing circuitry is further configured to: scaleeach dequantized residual sample of the set of dequantized residualsamples using a dequantized value associated with each of the firsttransform coefficients; and reconstruct the current block using the setof scaled dequantized residual samples.
 22. The device of claim 16,wherein one or more pre-stored residual samples of the identified set ofpre-stored residual samples are stored at a fractional precision in theportion of the video data stored to the video data memory.
 23. Thedevice of claim 16, wherein the processing circuitry is furtherconfigured to scale each pre-stored residual sample of the identifiedset of pre-stored residual samples using a pre-defined factor.
 24. Thedevice of claim 16, wherein the processing circuitry is furtherconfigured to clip each pre-stored residual sample of the identified setof pre-stored residual samples using a pre-defined factor such that eachclipped pre-stored residual sample can be represented using four (4)bits or less of data.
 25. The device of claim 16, wherein to combine theset of pre-stored residual samples to form the combination of pre-storedresidual samples, the processing circuitry is configured to perform alinear combination of the set of pre-stored residual samples to form alinear combination of residual samples pre-stored to the video datamemory.
 26. The device of claim 25, wherein to perform the linearcombination of the set of pre-stored residual samples, the processingcircuitry is configured to calculate a weighted sum of the set ofpre-stored residual samples, at least in part by assigning a respectiveweight to each pre-stored residual sample of the set of pre-storedresidual samples.
 27. The device of claim 16, wherein to combine the setof pre-stored residual samples to form the combination of pre-storedresidual samples, the processing circuitry is configured to merge theset of pre-stored residual samples to form a single merged residualblock, and wherein the processing circuitry is further configured tooverwrite the set of pre-stored residual samples with the single mergedresidual block in the video data memory.
 28. The device of claim 16,wherein the processing circuitry is further configured to: derive asingle residual block using one or more pre-stored residual samples ofthe identified set of pre-stored residual samples overwrite the set ofpre-stored residual samples with the derived single residual block inthe video data memory.
 29. The device of claim 16, wherein to code thevideo data, the processing circuitry is configured to decode encodedvideo data, wherein the current block is an encoded block of the encodedvideo data, and wherein to code the current block, the processingcircuitry is configured to reconstruct the encoded block.
 30. Anapparatus for coding video data, the apparatus comprising: means foridentifying a set of pre-stored residual samples in a pre-stored portionof the video data; means for combining the set of pre-stored residualsamples to form a combination of pre-stored residual samples; means forapplying a first sign combination to transform coefficients of thecombination of pre-stored residual samples to obtain a first hypothesisreconstruction with respect to a current block of the video data; meansfor applying a second sign combination to the transform coefficients ofthe combination of pre-stored residual samples to obtain a secondhypothesis reconstruction with respect to the current block of the videodata; means for deriving respective cost functions with respect to thefirst hypothesis reconstruction and the second hypothesisreconstruction; means for comparing the respective derived costfunctions; means for selecting, based on the comparison, either thefirst hypothesis reconstruction or the second hypothesis reconstruction;and means for coding the current block using a sign predictionassociated with the selected one of the first hypothesis reconstructionor the second hypothesis reconstruction.